Impact Evaluation of the School Electrification and ... · PDF fileFaustino & Medina #iematters 6 . Objectives of the Evaluation ... NAT Score t-value p-value Filipino 1.476 0.140

Impact Evaluation of the School Electrification and Education (SEEd) Project of the Alliance for Mindanao Off grid Renewable Energy

Program (AMORE): A Utilization of Propensity Score Matching (PSM)

Nelson C. Faustino, PhD, Former M&E Specialist, SEEd-AMORE

Michael Arieh P. Medina, PhD, Faculty, CFES, Central Mindanao University

Presented at the “Making IE Matters Conference” ADB Headquarters, Metro Manila, Philippines September 3-5, 2014

Outline

• Background of the Study

• Objectives

• Methods

• Results and Discussions

• Conclusions

• Recommendations

Faustino & Medina #iematters 2

Background of the Study

• Out of ~5,600 secondary schools:

– 67% of schools scored within the “lower average” range in the National Achievement Test (NAT)

– ~1/3 made it to the upper score ranges

– Only ~1% were rated superior

• Source: DepEd Order No. 72 s. 2011



• The Philippines ranked 5th out of 8 ASEAN countries

• Last in rank in terms of quality of Math and Science education and with the capacity for innovation.

– Global Competitiveness Report, WEF (2011)

Faustino & Medina #iematters 4 https://www.youtube.com/watch?v=z4bEkzwc0w4

https://www.youtube.com/watch?v=z4bEkzwc0w4


• …MULTIMEDIA can improve learning and retention of material presented during a class session or individual study period, as compared to “traditional” lectures or study materials…

– Ludwig et al. (2004)


The SEEd-AMORE Program

• USAID-funded in collaboration with SunPower Foundation, Intel Phils., and DepEd.

• Access to RE (solar) by off grid rural schools

• Provision of multimedia educational materials

– Improving English Language Teaching and Learning in Mindanao Program (Real World Productions)

– e-Media Program (ABS-CBN Foundation)

– Knowledge Channel


Objectives of the Evaluation

• Determine the influence of the SEEd-AMORE program on the educational performance of student beneficiaries (NAT Score).

– Compare the NAT Scores of student beneficiaries and non-beneficiary students

– Determine the degree of influence of the extent of ETV utilization on the NAT Scores of student beneficiaries.


Methods

• Survey of 409 project beneficiaries and 534 non-beneficiaries.

• Propensity Score Matching (PSM)

• t-test for independent groups

• Multiple Linear Regression (MLR)


Results and Discussions

• Unmatched Sample:

• Beneficiaries = 409, non-beneficiaries = 534

NAT Score Beneficiaries Non-Beneficiaries

Filipino 62.57 60.81

Math 67.85 62.65

English 62.50 61.56

Science 63.74 64.34

HeKaSi 66.80 63.11

MPS 64.69 62.49 Faustino & Medina #iematters 9




NAT Score t-value p-value


Math 3.187 0.001

English 0.678 0.498

Science 0.487 0.626

HeKaSi 2.811 0.005



• propensity score matching is a methodology used in attempting to provide unbiased estimation of treatment-effects.

– Rosenbaum & Rubin (1983)



• Step 1. Binary Logistic Regression Analysis

– In order to identify significant sources of bias

Covariates B Wald p-value Exp(B)

4 Ps Program (Non-Beneficiary) -0.565 14.850 0.000 0.569

Ownership of House and Lot -0.105 4.299 0.038 0.900

No. of Appliances at Home 0.310 24.514 0.000 0.733

Home Electricity Source (Off Grid) 0.398 4.486 0.034 1.488

Honors Received (No Honors) -0.463 10.187 0.001 0.629

α=0.05 Faustino & Medina #iematters 12

Description of the PSM (Nearest Neighbor) Methods

Method Replacement Caliper Match Ratio

1 With Without 1:1

2 Without Without 1:1

3 With 0.2 1:1

4 Without 0.2 1:1

5 With 0.2 1:2

6 Without 0.2 1:2

7 With Without 1:2

8 Without Without 1:2



• Step 2. PS Matching (Nearest Neighbor)

8 Matching Methods

Sufficient Units for Methods

1,2,3,4,5 & 7

Not enough Control Units for Methods

6 & 8

Significant Imbalance

for Method 2

Most Improvement in Balance for

Method 4

PS Matching Overall Balance Multivariate Imbalance







Math 3.187 0.001

English 0.678 0.498

Science 0.487 0.626

HeKaSi 2.811 0.005



• Matched Sample (Method 4):




Math 4.764 0.000

English 1.750 0.081

Science 1.752 0.080

HeKaSi 3.435 0.001


Results and Discussions • Multiple Linear Regression Analysis

Dependent Variable = NAT MPS (N=409)

Factors B β t-value p-value

(Constant) -120.48 -5.1 0.000 No. of Major Appliances 3.323 0.275 6.318 0.000 Attendance (%) 0.445 0.191 4.455 0.000 Parents are 4P’s members 9.377 0.276 6.481 0.000 Average School Grade (%) 0.928 0.171 4.024 0.000 Nutritional Status 4.133 0.105 2.479 0.014 Sineskwela 9.651 0.091 2.172 0.030 Mathtinik 9.699 0.115 2.398 0.017 Bayani 7.148 0.115 2.335 0.020


Conclusions

• The SEEd-AMORE program was able to influence the performance of student beneficiaries in terms of the NAT MPS

• The extent of utilization of multimedia learning materials (Sineskwela, MathTinik, and Bayani) influences the NAT MPS of student beneficiaries

• Other essential factors that influences school performance are: family income, attendance, 4P’s, School Grade, and Nutrition.


Recommendations

• Replication of the SEEd-AMORE program in other geographical regions.

• Intensification of the utilization of multimedia learning materials in the basic education curricula.

• Technical assistance to teachers in adapting to technology.

• Further evaluation of other programs related to the improvement of student performance in rural schools (e.g. 4P’s, Nutrition, etc.)


Literature Cited in this Presentation

Alliance for Mindanao Off Grid Renewable Energy (AMORE) Program (2011). School Electrification and Education (SEEd) Manual.

Tubeza, P. (2011). DepEd seeks higher test scores for RP students. Retrieved July 11, 2012 from http://newsinfo.inquirer.net/inquirerheadlines/nation/view/20100108-246133/DepEd-seeks-higher-test-scores-for-RP-students

Ludwig, T. et al. (2004). Using Multimedia In Classroom Presentations: Best Principles. Prepared for the Society for the Teaching of Psychology Pedagogical Innovations Task Force December 2004.

Rosenbaum, P. & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70 (1): 41–55. DOI:10.1093/biomet/70.1.41.

World Economic Forum (2011). Global Competitiveness Report


Acknowledgement

• University of Southeastern Philippines (USeP), College of Governance and Business (CGB)

• Central Mindanao University (CMU), College of Forestry and Environmental Science (CFES)

• Rec E. Eguia, PhD (USeP-CGB)

• Rural Off Grid Schools in Mindanao

• United States Agency for International Development (USAID) – SEEd-AMORE Program

• ADB, 3ie, PIDS


Daghang Salamat

Maraming Salamat

Thank You


Contact Details: Website: http://gradschooljungle.blogspot.com Email: [email protected], [email protected], [email protected]

http://gradschooljungle.blogspot.com/

Documents

Impact Evaluation of the School Electrification and ... · PDF fileFaustino & Medina #iematters 6 . Objectives of the Evaluation ... NAT Score t-value p-value Filipino 1.476 0.140