1

2

Context of Evaluation

Technology can be applied in the classroom

(and the community) in countless ways, and

with numerous benefits.

– The flexibility of technology presents both

opportunities and challenges.– Evaluation must remain focused on measuring and

monitoring the use of technology to fulfill the program’s goals.

3

Program Goals, Vision, Mission

Provide Opportunity for underprivileged children in India to enhance their learning through the use of Digital Technology

o Increasing Economic Potentialo Promoting Global Camaraderie

Future AIF/DE plans:o Expansion of program to additional schools.o Increase utilization of Internet Learning Centers

(ILCs) through community involvement.o Focus on government schools & the most needy

areas.

4

Current Measurements

Quantifiable Metrics:o 55 centerso 1000 teacherso 25,000 high school students trained in basic computer

and internet skills.o Almost 10 schools have relationships with the local

community to support sustainability and help cover costs.

o 39 Schools with Websiteso 100 teacher projectso 70 student projects

5

DE Benefits to StudentsAIF-Reported Benefits of the Digital Equalizer program include:• Considerable improvement in academic performance of slow learners.• The Internet motivates students to learn English faster.• Increased confidence & skills to be successful in life.

Our visits to 9 schools, including interviews and interactions withstudents, teachers, and school management, have led us to believe these claims are credible.

In addition, an overwhelming amount of 3rd party researchsupports the value of Technology Aided Learning (TAL).

6

Independent Research on TAL

Sivin-Kachala, J. (1998).

Report on the effectiveness of technology in schools, 1990-1997.  • Includes a review of 219 research studies from 1990 to 1997. • Students in technology rich environments showed increased

achievement:– In all major subject areas

– Among all different age groups

– Among both regular and special needs children

• Students attitudes toward learning and their own self concept improved consistently when computers were used for instruction

7

Additional Research on TALKulik, J.A. (1994). Meta-Analytic studies of findings on computer based

instruction. In E.L. Baker, and H.F. O’Neil, Jr. (Eds.). Technology assessment in education and training. Hillsdale, NJ: Lawrence Erlbaum.

Author Studies

Analyzed Percentile Gain

over Control Group

Bangert-Drowns, J, Kulik, & C. Kulik (1985) 51 10

Burns & Bozeman (1981) 44 14

Cohen & Dacanay (1991) 38 18

Hartley (1978) 3 16

Fletcher (1990) 28 19

C. Kulik & J. Kulik (1986) 119 11

C. Kulik, J. Kulik, & Schwalb (1986) 30 15

J. Kulik & Bangert-Drowns (1985) 44 16

Niemiec & Walbert (1985) 48 14

Roblyer (1988) 82 12

Schmidt, Weinstein, Neimiec, & Walberg (1985) 18 22

Willett, Yamashita, & Anderson (1983) 11 9

Total studies analyzed 516 Weighted Average Improvement 13.6 %

8

Additional TAL Benefits

McKinsey Report: Connecting K-12 schools to the Information Superhighway http://www.uark.edu/mckinsey

• “In a review of 254 controlled studies, researchers found that CAI helped students to learn 30 percent faster than students receiving traditional instruction.”

• “The use of computers seems to have even greater effects with low achieving and remedial students.”

• It seems clear that the impact on student motivation levels is significant. The Christopher Columbus School, which extensively uses networked computers, has its district's best attendance record for both students and faculty. In the Carrollton District, the dropout rate declined from 19% to 5% after deployment of technology.

9

Summary of Benefits

10

ChallengesAlthough the success of Technology Aided Learning is well documented, and the

value of the Digital Equalizer Program is clear to anyone with the opportunity to visit AIF schools, the need for ongoing assessment and evaluation remains.

More refined assessment and measurement will:– Increase effectiveness. – Provide assurance to donors, partners, and participants that the program is meeting

its goals.

Evaluation tools should also monitor progress and performance in areas where obstacles are expected.

– Motivation of Staff, Teachers in Government Schools.– Parental Support.– Utilization of ILCs during after school hours. – Sustainability:

• Skills development, continued training.• Financial costs of ILC operation and technology renewal.

11

Current DE program

Year One

• Detailed training plan• Standardized teacher training curriculum• Standardized review and assessment• Some flexibility incorporated into goal setting and

assessment

Year Three

Year Two

12

Learnings from School visits• Schools are unique

– Students, Teachers, Facilities, Finances.

• Each school has different needs and challenges:– More computers, training, support.

– Financial sustainability.

– Teacher or Management motivation

The DE Program must be customized to fit school needs

13

Digital Equalizer Program

IdentifyingSchools

Categorize School

Establish Objectives

Evaluate Explore

Sustainability

14

Identifying School

IdentifyingSchools

Categorize School

Establish Objectives

Evaluate Explore

SustainabilityIdentifyingSchools

15

DE – Identifying Schools

The selection of schools that are best positioned to benefit from TAL may be the most critical factor determining the success of the Digital Equalizer Program.

Our research and observations suggest the following key considerations:– School’s commitment to training, participation, and integration of

technology to improve the school. – Enthusiasm of principal and teachers.– Availability of DE coordinator (location).– Issues surrounding vernacular and English medium schools.

We recommend tracking selection process success, failures, and evaluating whether criteria being used are appropriate predictors of DE program success.

16

Categorize School

IdentifyingSchools

Categorize School

Establish Objectives

Evaluate Explore

Sustainability

17

DE – Categorize Schools

To efficiently customize the DE program foreach school, it is important to understandeach school’s needs.

By categorizing a school:– Comparisons between schools in similar situations

become more meaningful. – Different objectives, measurements, and expectations

are appropriate for schools in different categories.– DE coordinators can more easily manage the schools

and prioritize DE resources and attention.

18

School Categories

• Financial Position of School– Type of institution (Public, Private, Government)– Economic class of students’ families (ability to pay computer fees)

• English or Vernacular Medium– Vernacular schools face different challenges, but also possess enormous

opportunities to use the Internet and technology to teach English.

• School Size and Infrastructure– A lab 10 unit computer lab cannot serve 3000 students (plus teachers) in the

same way it serves a 300 student school.– Schools which already have some computer technology, or acquire additional

computers, will have different capabilities.

• Post Graduation Career Expectations– The Internet and computers can be best used in different ways, depending upon

whether students plan to continue their education, enter the workforce, etc.

• Boy, Girl, Coed, and Special Needs Education

19

Establish Objectives

IdentifyingSchools

Categorize School

Establish Objectives

Evaluate Explore

Sustainability

20

Establish Objectives

One group of consistent objectives and measurements across all schools:– Establishes minimum standards.– Can be used to assess and communicate the impact of the DE

program.

– Ties each school to the AIF/DE Mission. A second group of objectives and measurements that

are unique to the school is also critical:– Allows school to participate in setting expectations and

determining how to best adapt technology to its specific needs. – Since schools will be evaluated on the basis of meeting both

sets of objectives, this process allows comparison of schools with similar resources, abilities, and obstacles.

21

School and Program Objectives

As part of the AIF-DE Training Guide, the Implementation Plan (Phase 1, Units 1 and 2) provides Coordinators the opportunity to set both sets of Objectives.

• Coordinators should agree with School Management on specific objectives and goals which are measurable and in line with the core objectives of the DE program.

• As part of the objectives agreement process, schools should also be informed of (or cooperatively determine) the measurements and evaluation methods that will be used to track progress toward these objectives.

• Schools should be encouraged to set challenging goals, and should not fear punishment for failing to reach a goal that is set very high relative to the goals set by similar schools.

22

Establishing Common Objectives

Common Objectives should tie to the AIF/DE mission:– Access to Digital Technology– Enhanced Learning– Expanded Economic Opportunities– Global Camaraderie

Sample School Objectives might be:– Provide each student X hours of technology instruction– Provide X hours per week of unstructured ILC availability.– Boost student subject area test scores X percent– Increase number of students attending +2 by X percent– Provide every student in class 10 the opportunity to correspond

by email with a student in the United States.

23

Establishment of Unique Category/Objectives

Unique Objectives should tie to the specific school:– Is attendance or enrollment a problem?– Is financial sustainability a reasonable goal for the school to

accomplish on its own?– Is the school an English medium or vernacular school?

Sample Objectives might be:– Increase attendance to X percent.– Creation of a student fee program which covers certain digital

technology costs.– Increase English proficiency test scores by X percent.

24

Evaluate

IdentifyingSchools

Categorize School

Establish Objectives

Evaluate Explore

Sustainability

25

Evaluation

• Evaluation of individual schools must tie to both sets of agreed upon objectives.

• The AIF-DE Training Guide currently includes assessment templates for teachers, students, schools, and the DE program.

– Comprehensive checklists– Primary weakness is that the grading scale is subject to interpretation, making

assessments completed by different individuals impossible to compare. Also may lack credibility to outside constituents such as donors.

• However, the AIF-DE Training Guide also includes opportunities and guidelines that will allow collection of important information that can be used for quantifiable, unbiased evaluation (See Attachment 1).

• Additional data collection will aid evaluation process (See Attachment 2)

• The evaluation should be a two way process where all constituents get their roles and responsibilities reviewed. A tool for schools to evaluate DE coordinators should be developed with input from facilitators.

26

Evaluation Questions

Question Set ACheck points and success factors that are common in all schools

•Basic School Info•Teacher Training

•Curriculum and Academics•Facility Utilization and Reliability

•School Management commitment and support•Financial Tracking

Question Set BCheck points tailored for each school

27

Evaluation – Ongoing and Iterative

Pre-programEvaluation

YEAR 1 Evaluation

YEAR 2 Evaluation

YEAR 1 YEAR 2 YEAR 3 YEAR 3 Evaluation

•Initial evaluation provides baseline measurements that are needed for later comparison and that are critical to showing results of the DE program.•First, Second, and Third year assessments match each years activities and objectives, so there is some variation in the evaluation. •Plans for methods of continued evaluation and assessment of schools after the three year period should begin immediately.

28

Explore Sustainability

IdentifyingSchools

Categorize School

Establish Objectives

Evaluate Explore

Sustainability

29

Sustenance Requirements: Money and Skills

For a school to sustain an ILC, it needs both:– Staff with the technical skills needed to run a lab– Financial resources to pay costs associated with normal use and maintenance.

All DE schools should possess staff and teachers with the skills needed to run and sustain an Internet Leaning Center by end of 3 years.

We are uncertain whether all government schools will be capable of finding the money to pay for connectivity, hardware and software upgrades, maintenance, and other fees that will be associated with sustaining the ILC over the long-term.

Although partnering with corporations and other community groups for sponsorships (adoption) of schools is one of the best options, the risk that such outside support will be focused on primarily the most successful model schools exists.

30

Summary of Costs

REQUIREMENTS INITIAL COSTS (in rps) RECURRING COSTS (in rps)

Hardware FREE – 10 computers/school, 1 printer, 1 scanner, 1 digital camera, 2 ups, LAN, ISDN connectivity (donated by Wipro)

Need additional computers for 1 child: 1 computer, CD burner.

5 year replacement cycle for most hardware.

Software FREE – Windows 98, Office 98, Publisher Need additional software:•O/S upgrades•ileap translation software•Anti-virus software

Teacher Training FREE - provided through Intel’s Teach to the Future Need additional training:•Technical support•Additional software training

Furniture (computer desks, chairs, exhaust fans etc.)

19500 rps. 50 chairs @ 390rps each Need additional workstations for 1 child: 1 computer

Electricity 2000rps/mos.

Internet Connectivity FREE – one year included Must spend 3500 – 4500rps/mos after first year

Supplies (paper, toner, disks, cds, etc.)

2000rps/mos

Hardware maintenance FREE – first 3 years covered under warranty 250rps/call

OPERATING BUDGET 8,000 – 10,000 rps/month

31

Recommendations: Overall ProgramAll of our recommendations should be viewed in the context of

how deeply impressed we are with the success of the DE program, the hard work of the staff, and and our awareness that in the short time we have had here we cannot expect to know more about this program than the AIF-DE team.

• “AIF DE program,” “Schools Online,” “SOL” and “Digital Equalizer” identity should be further consolidated.

– Provides maximum value and clarity for donors who are interested in participating in program

• Financial assistance, rewards in kind, certificates, other motivating actions to keep ILCs open and available to students for additional hours once schools would normally close should be considered.

– Spending by AIF is certainly justified to increase ILC utilization and maximize the value of the centers to students, teachers, and the community.

32

Curriculum Recommendations

• Continued Focus on teacher professional development. – No need focus on 100% teacher training if some staff is near retirement. – Those resistant to computers should be paired with a computer trained

teacher– Teachers with a desire to learn beyond AIF/DE training should be

directed to Online Training or other development opportunities.• Create a national digital trust for content development

– Portal development – Partnerships with other organizations to share content.– Share what works between schools. Email discussion lists, other forms

of knowledge sharing. • Training curriculum should be flexible and adaptable to

different schools• Emphasize that we are concentrating on learning through

computers rather than learning about computers.

33

Infrastructure Recommendations

• The current machines may be obsolete in three to five years.– Expensive upgrades of software and hardware will be required if the Digital

Divide is to be eliminated. – AIF should verify arrangements with WIPRO, or make other plans needed to

ensure sustenance after 3 year program.

• Explore other technology options that can broaden impact of digital technology:

– Computers outside the ILCs, or scale the DE investment based on number of students in grades 7 to 10.

– Digital Projectors for large classroom/auditorium.– Traditional overheads, other types of study materials that can be constructed with

the help of the Internet, PCs.

• "Professionalize" technical support. – This could either be a dedicated person in the school or a key contact at the

hardware or software company. – Having one facilitator responsible for training, running the lab,and all technical

support (in addition to teaching responsibilities) may not be reasonable