Evaluation of The Recycle Bowl

8/9/2019 Evaluation of The Recycle Bowl

1/67

Recycle-Bowl Competition - Program Evaluation

Cinthia Josette Arvalo

Blint Pet

Agustina Suaya

Morgan Taylor

Spring 2013

T r a c h t e n b e r g S c h o o l o f P u b l i c P o l i c y a n d P u b l i c A d m i n i s t r a t i o n

M a s t e r o f P u b l i c P o l i c y C a p s t o n e : S p r i n g 2 0 1 3


2/67

2

Table of Contents

Acknowledgements ....................................................................................................................................... 3

Executive Summary ...................................................................................................................................... 3

1. Introduction & Background .................................................................................................................. 4

2. Literature Review .................................................................................................................................. 6

3. Data & Methodology .......................................................................................................................... 10

Data ......................................................................................................................................................................... 10

Methodology ........................................................................................................................................................... 13

4. Descriptive statistics ........................................................................................................................... 17

5. Analysis............................................................................................................................................... 22

Question 1: Effect of Haulers and Self-haulers on Recycling Rates .......... .......... ........... .......... ........... .......... ......... 22

Question 2: Effect of the Recycle-Bowl Competition ............................................................................................. 28

Question 3: Analysis of data collection methods used by KAB .......... .......... ........... .......... ........... .......... .......... ...... 30

6. Limitations .......................................................................................................................................... 32

7. Final Comments .................................................................................................................................. 36

8. Tables and figures ............................................................................................................................... 38

Recommended Updates to Recycle-Bowl Competition Surveys ......... ........... .......... ........... .......... ........... .......... .... 43

9. References ........................................................................................................................................... 51

10. Appendix ......................................................................................................................................... 53


3/67

3

Acknowledgements

The authors would like to acknowledge Kelley Dennings and Keep America Beautiful for giving

us the opportunity to work for them on this evaluation. Ms. Dennings provided the data and

contact information for the recycling coordinators, without which this project would never have

taken place. The authors would also like to acknowledge the recycling coordinators who sent us

pre-competition data for our analyses. The authors would also like to acknowledge Professors

Kathryn Newcomer and Dylan Conger for their advice on the evaluation process and data

analysis. The authors would also like to acknowledge both Professor Elizabeth Rigby and Gene

Carpenter for their guidance and assistance throughout the project.

Executive Summary

This document is the product of the Capstone project completed by four graduate students of the

George Washington Universitys Master of Public Policy program for Keep America Beautiful

(KAB). The study evaluates the effects of KABs schools recycling competition, the Recycle-

Bowl, a contest that reaches out to approximately one million students, and engages schools and

their community in recycling.

The evaluation focuses on three research questions: (1) Do schools that self-haul recycle morethan those that work with hauling companies? How important is this factor compared to other

school factors in shaping recycling rates among participating schools?; (2) Does the competition

meet its objective of increasing recycling rates in participating schools?; (3) What additional data

could KAB collect in the survey to better assess the programs impact, particularly in the long-

term?

The study worked with data provided by KAB about schools that registered and competed.

Additionally, the authors collected pre-competition data about weights recycled by schools andabout the proportion of students eligible for free or reduced price lunch, a proxy for income

levels. The final dataset, after cleaning and excluding some observations, contained 1,118

schools. Pre-competition data were collected for about one quarter of participating schools

(277), and for 193 non-participants. From these data, two datasets were generated: one cross-

sectional, and one employing a panel design.


4/67

4

The study concludes that:

(1)There is no statistically significant difference between the per-capita weight recycled of

schools that self-haul and per-capita weight recycled of schools that have a commercial

hauler, after controlling for all available observed characteristics.(2)The Recycle-Bowl Competition had a positive effect on competing schools of increasing

their recycling rates by 0.44 pounds per capita during the competition, relative to schools

that did not participate in the program in 2012. However due to the limitations to the

analysis, the result should not be interpreted as a causal effect.

(3)Some changes need to be implemented in the registration and results reporting survey

sent out to schools during competition, which will assist in the data analysis of future

evaluations.

Due to many threats to validity, the conclusions of the dataset have their limitations. First of all,

conclusions have no generalizability outside the sample. Second, omitted variable bias and

selection bias pose threats to internal validity. Due to these threats, causality cannot be

determined.

We are hoping that our work has contributed to better understanding of the effect mechanisms of

Recycle-Bowl, and to designing a data collection procedure that allows for future impact

evaluations with greater internal, and potentially external, validity.

1.

Introduction & Background

The Master of Public Policy degree program at George Washington Universitys Trachtenberg

School of Public Policy and Public Administration concludes with a pro bono Capstone research

project for an external client. The present study is the product of a Capstone project performed

by the authors for the non-profit organization Keep America Beautiful (KAB) during January-

April 2013.

KAB is engaged with more than 1,200 affiliates and participating organizations and with

millions of volunteers to develop various programs with the aim to improve the community

environment. Keep America Beautifuls mission is to engage individuals to take greater


5/67

5

responsibility for improving their communitys environment.1 Its mission includes offering

solutions that create clean, beautiful public places, reduce waste and increase recycling,

generate positive impact on local economies and inspire generations of environmental stewards.

KAB offers programs through public-private partnerships that help individuals and organizations

engage to take greater responsibility for improving their communitys environment. One of

these programs is the annual Recycle-Bowl Competition.

The Recycle-Bowl Competition is a national recycling contest that started in 2011 for K-12

schools occurring each fall semester. In 2012, the competition took place from October 15 to

November 9. Recycle-Bowl is designed for teachers, student green teams and facility managers

to engage students and the community in recycling.2

In 2012, KABs Recycle-Bowl reached

approximately 1 million students.

In addition, it is important to mention that Recycle-Bowl has two categories of winners

announced each year: schools that participate in the competition for the cash prize and only

count recycling material within the school (called School-to-School in this study), and schools

that also collect recycled material from the public (called Community-to-School in this study)

and do not compete against schools that only recycle their own waste. This study will perform

different analyses for these two categories.

Objectives of the Research

This Capstone projects objective is to provide the client, KAB, with useful information on the

Recycle-Bowl Competitions impacts, important factors that drive school recycling rates, and

recommendations on how to improve data collection in the future to better assess the programs

effects. KAB will benefit from this research project by having a deeper understanding of the

effects of the program, as well as information to shape their managerial decisions regarding how

to implement the program. Second, this information could allow KAB to document its current

program operation and effectiveness in order to retain and/or increase funding sources. Finally,

one of the implicit objectives of KAB is to be the leading organization in the field and to reach

more schools and students every year. Therefore, this study can provide KAB with information

1Mission Statement provided by Kelley Dennings from Keep America Beautiful.2http://www.kab.org/site/PageServer?pagename=programs_home


6/67

6

they can use not only to promote the engagement of the schools that are already in the

competition, but also to also promote the participation of new schools into the competition.

Specific research questions

1. Do schools that self-haul recycle more than those that work with hauling companies?

How important is this factor compared to other school factors in shaping recycling rates

among participating schools?

2. Does the competition meet its objective of increasing recycling rates in participating

schools? Is there also an increase in recycling in the long term?

3. What additional data could KAB collect in the survey to better assess the programs

impact, particularly in the long-term?

2. Literature Review

Overview

From previous literature regarding recycling habits, the following conclusions have been

extracted. First, policy context matters in patterns of recycling; local policies can fosterrecycling behaviors. Second, local drop-off and curbside policies show positive effects on

recycling for the entire community. Third, socio-economic factors also matter in recycling

behavior. The literature suggests that higher-income families are more inclined to recycle.

Fourth, recycling programs in other parts of the US and Canada that similar are to the one being

examined in this analysis have seen successful results. Finally, commitment, attitude, and the

amount of effort required all play an important role of forming the long-term recycling behavior.

The Role of Local Policies in Fostering Recycling

Local policies have an effect in fostering recycling. Some areas in the United States see more

recycling rates than others due to the presence of recycling regulations. Likewise, community

recycling programs, such as curbside recycling, increase recycling rates for those communities

that participate.


7/67

7

Almost 60% of the states have some kind of regulation that promotes recycling. All the states of

the Middle Atlantic and New England regions have at least one law that foster recycling.

However, in the South-East Central region and Plains region none of the states have regulations.

Regulations regarding recycling include Bottle Bill Laws, electronic waste laws, and newly

launched regulations to reduce the number of plastic bags.

Curbside programs, which have gained popularity in the last few decades, promote recycling

through the collection of recyclable items from the consumer (Jenkins et al, 2003). Some

programs are voluntary, but there are some instances of mandatory curbside programs. Jenkins

et al. (2003) used data from a survey to analyze the recycling participation of different

households. They find that two elements seem to be critical to improve the recycling of the

families: presence of a local drop-off recycling near the house and existence of curbside

recycling program in the area. A study by Reschovsky and Stone (1994) focusing on the use of

market incentives to encourage household waste recycling by pricing waste-disposal services

according to the quantity of waste generated, finds that curbside pickup had the greatest effect on

reported recycling behavior.

Maximizing and sustaining citizen participation in solid waste recycling programs depends upon

the programs design. Folzs (1991) analysis of municipalities shows that democratizing the

programs design and planning process had significant impacts on citizen participation, as didmandating participation in recycling. In addition, Folz mentions even voluntary programs

worked well when combined with other implementation strategies, such as curbside delivery, the

use of private contractors, public education programs, and the provision of free bins (Folz,

1991: 222).

Nyamwange (1996) includes several strategies for recycling program designs in her evaluation of

public perception of recycling programs: increasing the level of knowledge about recycling,

using effective channels to inform the community about recycling, increasing the convenience of

recycling by placing recycling containers in accessible locations, and getting input from the

public regarding changes that would induce full participation in recycling programs

(Nyamwange 1996). Likewise, Derksen and Gartrell (1991) show that communities with

structured recycling programs tend to recycle more than those without. Attitudes towards the


8/67

8

environment only affect the level of recycling for those communities with structured systems, as

attitudes cannot overcome the lack of community programs. Another important issue that the

literature suggests is that recycling programs seem to increase their effectiveness over time

(Jenkins et al, 2003).

Recycling regulations and community programs have grown in the past few decades, thus

increasing recycling rates. Individual participation in recycling can be fostered through the

provision of community recycling programs. However, the design of recycling programs and

educating the public about such programs are also important in increasing participation levels.

Socio-economic Factors and Their Relationship with Recycling Rates

Socio-economic factors also play a role in the recycling rate of a household. Granzin and Olsen

(1991) discuss what types of people participate in pro-environment activities, such as recycling.

While younger generations tend to have more concern for environmental issues, older

generations are more likely to actively participate in recycling. Likewise, individuals with

higher education levels tend to demonstrate more concern about the environment and new

conservation activities. Similar results were found during studies looking at the effect of income

on recycling, where higher income brackets tend to actively participate in recycling activities

more than less-affluent individuals. Jenkins et al. found that household income has a significant

and positive effect on intensity of recycling effort for newspaper only (Jenkins et al, 2003: 313).

Household size has also some effect on the recycling behavior. Increasing the number of

occupants of the average household by 1 person leads to a 3 percent increase in the probability

that the household will recycle over 95 percent of its glass bottle waste and a 2 percent increase

in the probability of recycling over 95 percent of yard waste (Jenkins et al, 2003: 313). In

addition, they observed that single-family dwellings show a higher participation in curbside and

drop-off recycling programs, something that was also found in previous studies (Granzin and

Olsen, 1991: EPA, 1994). People who recycle often view environmental issues as a whole-worldissue, thus increasing their recycling efforts. As shown through the literature, socio-economic

factors affect recycling rates. Individuals with higher education, and who are more affluent, tend

to recycle more than others.


9/67

9

Similar Programs to the Recycle-Bowl Competition and Their Achievements

Similar programs to Recycle-Bowl are being implemented in the U.S. and Canada, which share

experienced overall success in their competitions. The nonprofit organization Encorp Pacific

(Canada) created the School Recycling Program for beverage containers in British Columbiaschools. Cash prizes are given to schools that performed better in a year. The Cambridge

Department of Public Works (Massachusetts) has two recycling initiatives to promote recycling

in schools. RecycleCraze is a fourteen week recycling competition between the Cambridge

Public Schools, where winners receive a unique recycled metal trophy by a renowned artist. The

Rockin' Recyclers Award is an award to recognize the school with the best efforts to educate

and involve students in recycling and waste prevention. The winning school is given public

recognition. Mecklenburg County Land Use & Environmental Services Agencys solid waste

division staff conducted a study of 24 local government recycling and waste diversion programs

to identify program elements that offer successful diversion of solid waste from landfills. They

conclude all school programs vary widely among local governments in terms of employee

dedication, program type, and sponsorship. The overall success of these programs, marked by

high recycling rates in the schools, shows that such programs can potentially shape behavior.

Factors that Affect Long-Term Behavioral Change in Recycling

Certain factors affect long-term behavior change regarding recycling, including ownership of

commitment and acknowledgement from others. However, behavior change can be difficult to

achieve. Carlson (2001) examines whether social norms can resolve large-number, small

payoff problems, which require heterogeneous groups of individuals with no real connection to

one another to change their behavior for little or no economic gainenvironmental matters (such

as waste recycling) for example. Carlson concludes that norm creation or norm management in

itself is insufficient for solving such problems if the behavioral change is inconvenient or

requires significant effort. Thus, targeting behavioral change requires either the amount of effort

needs to be decreased or financial incentives be put in place, rather than just strengthening social

norms. Shultz and Oskamp (1996) also researched the role of effort in behavioral change, and

found that larger effort required results in attitude becoming a stronger predictor of behavior

change. The effect of attitude on recycling was also examined by Biswas et al. (2000), who


10/67

10

found that attitude has a significant effect on recycling behavior. Thus, if attitudes can be

changed, behavior will follow accordingly.

Baca-Motes et al. (2013) examine the effect of commitment on behavior change. They

conducted an experiment to determine whether hotel guests commitment to practice

environmentally friendly behavior during their stay motivates environmentally friendly behavior.

They found that when guests made a brief, but specific commitment at check-in, and received a

lapel pin to symbolize their commitment, they were over 25% more likely to hang at least one

towel for reuse; this increased the total number of towels hung by over 40%. Their experiment

proved that a symbolic commitment with no financial, moral or any other potential consequences

can significantly change the behavior of individuals, especially if the commitment is publicly

announced, because individuals manage their image via social signalingthey behave in ways

that communicate to others what kind of a person they are (Baca-Motes et al., 2013, p. 1071.)

While behavior change prove to be difficult to full achieve, increasing the feeling of commitment

to others has shown to increase behavior change regarding recycling.

The findings from the literature review shaped the methodology for evaluating the effect of the

Recycle-Bowl Competition. Controls for perception of the recycling program and the physical

location of the school (by region, as well as urban, rural, and suburban areas) have been included

per the results found in the first section of the literature review. From the second section,controls were created for income-level of the school and type of school (public, private, or

charter) to control for socio-economic factors which may affect the rate of recycling. A more in-

depth analysis of the methodology follows in the next section.

3.

Data & Methodology

Data

To answer the research questions, the authors of this paper drew from a range of data sources

which included a registration dataset of schools that registered to compete in the Recycle-Bowl

Competition (provided by KAB), a dataset containing competition results of the Recycle-Bowl


11/67

11

Competition reported by competing schools (provided by KAB)3, and information on the

percentage of schools population that are eligible for subsidized lunch from the National Center

for Education Statistics. In addition to these, the authors collected pre-competition recycled

weights data from recycling coordinators. From these data, two datasets were generatedone

cross-sectional and one employing a panel designas described below.

Cross-sectional Data

KAB provided two datasets with information on school characteristics (from the registration

dataset) and recycling weights (from the results dataset) of participating schools in 2012. The

datasets contained registration and results information separately with a unique identifier

available only for the results dataset. Information on 1,575 schools that registered for KAB

Recycle-Bowl Competition in 2012 (661 schools registered by non-coordinators and 914 schools

registered by recycling coordinators) was available in the registration dataset. Approximately

72% (1,138) of those schools reported the recycling results after the competition.

The information of the registration and results datasets was combined by matching the schools

names within states and by assigning the same unique identifier to those schools in the

registration dataset. This matching resulted in a single dataset with school characteristics and

recycling weights of 1,125 schools that participated and reported results (13 schools could not be

matched. There is no reason to believe that excluding these 13 cases changed the results since

they are assumed to be missing at random.)

Out of the 1,138 schools, 21 were excluded from the analysis because their reported competition

results could not be matched with the registration data (13 schools), or they were considered

outliers according to their reported per capita recycling weights (7 schools).4 Thus, the analytic

sample for the cross sectional analysis is 1,118 schools that participated in the Recycle-Bowl

Competition. (SeeTable 2.)

3Schools complete registration application and a reporting results survey, these came in the form of two sep arate

datasets.4A school was classified as outlier if its reported per capita weight recycled during the competition was more than

3 standard deviations higher than the mean per capita weight recycled. Thus, the study excluded schools with a

higher than 80.54 per capita recycling weights.


12/67

12

In addition, in order to include an income level control, zip codes were used to match schools

with information on the percentage of students eligible for free or reduced price lunch, a

common proxy for the income level of students in the school. These data were obtained from the

National Center for Education Statistics Common Core Dataset (NCES 2012). Consequently,

133 additional schools were excluded because they could not be matched to this administrative

data, leaving an analytic subsample for the regression analysis that includes income of 9845

schools who participated in Recycle-Bowl in 2012.

Panel Dataset

The panel dataset includes information from the cross-sectional dataset mentioned above, and

information on monthly pre-competition recycling weights from some schools. The authors of

this paper elicited additional information by contacting recycling coordinators by email to collect

monthly amounts from before the competition. The pre-competition data was obtained from a

small subset of recycling coordinators that were willing and able to provide that information in

the given time frame, and therefore is from a non-random subset of schools. In some cases there

is information available on a couple of months previous the competition, and in other cases there

are more data points in time. There was a positive side-effect to the data-collection campaign.

As some coordinators sent data on all of the schools under their supervision, the authors had

access to amounts recycled in schools that did not participate in the competition, which is used as

a comparison group (one that was not randomly selected) for the analysis. The authors of this

paper managed to obtain information from 12 recycling coordinators who provided pre-

competition data for 441 schools (270 schools that participated in the Recycle-Bowl

Competition, and 171 schools that did not participate in the competition and are used as a

comparison group). It is important to note that this dataset contains information only of schools

with recycling coordinators in 11 states6, and therefore the analysis might only be representative

of that sample. In addition, the information of the subsample comparison group dataset of 171

schools that did not participate in the competition came only from 3 recycling coordinators of

only 3 states: Indiana, North Carolina and Nebraska. Thus, conclusions that result from the

5When the income variable is added to the hauler regression for the whole sample, the sample drops to 979.6AL, CT, FL, IN, MN, NC, NE, PA, RI, SC, and TX.


13/67

13

panel data analysis that uses this comparison subset might not be generalizable to participating

schools in other states.

Methodology

In order to address the research questions, this study performs a quasi-experimental design

analysis. Given that the task is to assess the impact of a program that has already been

implemented, conducting the analysis with the gold standard experimental design is not

possible for two main reasons. First, schools that participate in the competition were not

randomly selected from the population of schools. Second, treatment (that is participation in

the program) was not randomly assigned to schools that were selected for the program. Thus, the

analysis will have to be conducted with extreme caution due to the number of threats to both the

internal and the external validity of the results and conclusions. Threats to validity will be

thoroughly discussed throughout the analysis.

First QuestionCross-sectional Hauler Analysis

The methodology for the first topic of this study (Do schools that self-haul recycle more than

those that work with hauling companies?) is described in this section.

The data used for this analysis, provided by KAB, includes data from the registration and results

datasets of the 2012 Recycle-Bowl Competition (1,118 schools). In order to assess if schools

that self-haul recycle more (in recycling per capita terms) than schools with commercial haulers,

the chosen method is regression analysis. The methodology tries to make sure the analysis has

ruled out other possibilities that could explain the differences in recycling weights per capita

between the two groups. Nevertheless, it is important to mention that this methodology has to

deal with threats to internal validity, such as selection bias.

The authors perform a cross-sectional regression analysis, controlling for school characteristics

in two specifications:

1. Bivariate regression analysis that compares recycling weights per capita between self-

hauler schools and schools with commercial haulers.

2. The second (full) specification controls for all available characteristics of schools:

whether they have recycling coordinators or not, the weighting report method (e.g. actual


14/67

14

weight from hauler, actual weigh from house scale, volume-to-weight estimates), school-

level characteristics such as type of school (public, private, charter), zone (urban, rural,

suburban), and level (elementary, middle, high), the income level of the school

(percentage of reduced price and free lunch eligible students is used as a proxy), and

finally it controls for regional/geographical characteristics.

The regression equation for the first specification takes the following form:

(1) Weightpcs= 0+ 1SelfHaulers+ s

Where: Weightpcs= the recycle weight per capita of schools

SelfHaulers = dummy variable that equals 1 if the school self-hauls the recycling material

and equals 0 if the school has a commercial hauler.s= the error term

The coefficient of interest is 1, which indicates if schools that self-haul compared to schools

with commercial haulers recycle more (or less). If the coefficient is positive and statistically

significant, then self-hauler schools would, on average, recycle more than schools with

commercial haulers, and vice versa.

The regression equation that includes all the controls takes the following form:

(2) Weightpcs= 0+ 1 SelfHaulers+ 2 Coords + 3 WMethods + 4Incomes+ Ss+ Rs+ s

Where: Weightpcs= the recycle weight per capita of schools

SelfHaulers = dummy variable that equals 1 if the school self-hauls the recycling material

and equals 0 if the school has a commercial hauler.

Coords= dummy variable that equals 1 if the school has a recycling coordinator

WMethods= a set of dummy variables that measure they weighting method used by the

school (5 weight method dummies. The reference group is actual weight from hauler)

Incomess= school income level proxy that measures the percentage of students that are

eligible to reduced and free lunch.

Ss= a vector of school characteristic controls (type, zone, and level)


15/67

15

Rs= a set of regional dummy variables (8 regional dummy variables. The reference group

is East North Central).

s= the error term

The coefficient of interest is 1, and its interpretation is similar to that in the first specification: it

indicates if schools that self-haul compared to schools with commercial haulers recycle more (or

less), holding all control variables constant. If the coefficient is positive and statistically

significant, then self-haulers would, on average, recycle more than those contracted with

commercial haulers, and vice versa.

Second QuestionPre-/-Post Comparison Group Design

In order to answer the question of whether there is an increase in recycling rates in schools that

participate in the Recycle-Bowl Competition, the methodological approach consists of a before-

and-after non-experimental difference-in-difference design with a comparison group using the

panel dataset for schools that competed in the Recycle-Bowl Competition and the comparison

group schools (441 schools in total7). Equation (3) is a Difference-in-Difference estimation with

school fixed effects8. The regression equations take the following form:

(3)Weightpcst= 0+ 1RBmonthst+ 2Interactionst+ s+ st

Where: Weightpcst= the recycle weight per capita of schools in timet

RBmonthst= dummy variable that equals 1 ifweight per capita data corresponds to

October or November 2012 (which are the months the Recycle-Bowl competition takes

place) and equals zero if data corresponds to September 2012.

Interactionst= is the interaction term product of the multiplication ofRecycleBowlst

(RecycleBowlst= dummy variable that equals 1 if school participated in the Recycle-

Bowl competition and equals 0 if did not participate in the competition) andRBmonthst

. It

is equal to 1 if data corresponds to participating schools during Recycle-Bowl

7 Outliers in weight per capita were excluded from the analysis8In addition, we added cluster controls to the model.


16/67

16

competition months, and equals zero if data corresponds to comparison group (non-

participating schools) or September 2012.

s= School fixed effects.9

Coefficient 1in equation (3) indicates the difference in average recycling weight per capita for

schools that did not participate between September 2012 and Recycle-Bowl months (October and

November), holding constant all school characteristics that do not change over time. The

coefficient of interest, 2, indicates a difference-in-difference estimation of how much

participating schools increased (or decreased) their recycling rates from September to

October/November compared to the change in recycling rates in the same time frame for non-

participating schools, holding constant all school characteristics that do not change over time. In

other words, this coefficient provides an estimate of the effect of the Recycle-Bowl Competition

on recycling rates.

The long term part of the question: Is there also an increase in recycling in the long term?

would require post-competition future data collection for several points in time, so that the long

term behavioral change of schools could be examined. This was not feasible in the available

timeframe of this study. Nevertheless, the results of this analysis will provide valuable insights

about the attitudes of schools toward recycling and potentially about behavioral change.

However, further quantitative analysis following up on this work will be required for morereliable conclusions.

Third Question - Improving the Recycle-Bowl Competition Survey

The third task in the evaluation is to give recommendations to Keep America Beautiful on how

they could improve the survey sent out to schools during the competition.

This task was performed last, based on the issues that were faced in answering the first two sets

of evaluations questions (especially data processing issues). The survey will have to be

improved based on the following three principles: (1) it will have to collect all necessary data for

further evaluations (including long-term impact evaluations); (2) it will refrain from collecting

9Controls for things that affect recycling rates and that differ across schools but that are constant over time for each

school, such as location, student/teacher population, infrastructure, etc.


17/67

17

unnecessary data in order to keep the workload of survey responders at a reasonable level, and to

increase response rates; and (3) the survey has to meet all criteria for a good survey, such as

phrasing questions that are as short as possible, that have simple unbiased vocabulary, it has to

be confined to just one issue per question, and it has to avoid the possible bias carried by the

structure of a survey.

All in all, a future survey will have to rely on the experience gained from this evaluation, and it

has to provide for future similar evaluations.

4.

Descriptive statistics

This descriptive section first presents some basic, univariate descriptive statistics of the dataset

of the 1,118 remaining schools, and then presents bivariate statistics based on the following 4

dummy variables: (1) whether or not schools had a recycling coordinator, (2) whether or not they

are self-haulers, (3) whether they participated in the School-to-School or in the Community-to

School competition, and (4) whether or not schools perceived the competition as having a

positive impact on their recycling weights.

Univariate Descriptive Statistics

Table 2in


18/67

18

Tables and figurespresents the basic univariate statistics of participating schools that were not

excluded from the sample.10

These 1,118 schools have a total school population of 893,033 (including teacher population). In

total, they recycled about 1,860 tons of materials during the competition, averaging a per capita

recycling weight of 5.2 per school. Recycling coordinators are present in 67% of schools, and

87% have external hauler agreements. A vast majority of the schools are public schools (over

90%), and they show an approximately even distribution in the proportion of students eligible for

free or reduced price lunch. Urban and suburban schools are also approximately equally

represented, while there are some more schools from rural neighborhoods in the sample (37%).

A majority of the schools (55%) perceived that the Recycle Bowl Competition had a positive

impact in their recycling weights.

Bivariate Descriptive Statistics

Table 3in

10A similar table in theAppendix (Table 11presents univariate statistics for the full set of variables.


19/67

19

Tables and figurespresents data based on whether competition results were provided by

recycling coordinators (67%) or not (33%). Differences between schools with and without

coordinators are important to consider since only schools with recycling coordinators were

contacted in order to obtain pre-competition data for a before-and-after comparison to answer the

second evaluation question (whether the competition increased recycling rates). Therefore,

coordinator/no-coordinator differences could bias the estimates based on data only from schools

with coordinators.11

There are statistically significant differences between schools with and without recycling

coordinators in a number of variables.12

It might be surprising that the average per capita weight

recycled is significantly higher in schools that do not have a recycling coordinator (no

coordinators: 6.77 pounds, coordinators: 4.44 pounds). This may be due to the fact that

coordinators take too good care of schools that, as a result, have little to do in the competition

and hence their performance is less conscious. This phenomenon is also reflected in the fact that

schools are more likely to have perceived the competition as having a positive impact on their

recycling weights if they do not have a coordinator (77.6 % vs. 43.4%).

At the same time, coordinators keep the data organized, so the percentage of schools with high

confidence in the measurement methods is significantly higher for the ones that work with

coordinators (84.4%) than for those that do not (69.9%).

The data show that schools that serve more affluent neighborhoods (the ones with lower

proportion of students on free and reduced price lunches) are less likely to have a recycling

coordinator. The data also show that there is a significantly larger portion of schools that self-

haul their recycling material among schools without coordinators (36.1% of schools without

coordinators are self-haulers, while only 1.9% of schools with coordinators are self-haulers).

Table 4 in

11This does not affect the cross-sectional analysis that assesses whether self-haulers recycle more than schools with

hauling services since all schools that reported results for 2012 are included.12

A similar table in theAppendix (Table 12presents coordinator-comparison statistics for the full set of variables.


20/67

20

Tables and figures compares descriptive statistics of schools that are self-haulers (146 schools,

13%) to those of schools with external hauler agreements (972 schools, 87%). Note that a

separate section of this analysis aims at answering the first evaluation question about the

differences in recycling rates between self-haulers and non-self-haulers.

There are two major and statistically significant differences between the two groups. First, self-

hauling schools are much more likely to have perceived the competition as having a positive

impact on their recycling weights. This may be related to the fact that schools with coordinators

have a less positive perception of impact of the competition on their recycling weights. As the

proportion of schools with coordinators is significantly higher among schools with hauler

agreements (75.9%) than among self-haulers (9.59%), the coordinator effect on positive

perception is (probably spuriously) captured as a hauler effect. Another plausible explanation for

this might be that if a school self-hauls students and teachers can regularly see the materials

recycled at the facilities of the school, and this might positively affect the perception of the

competitions effect. On the contrary, schools that have a contracted commercial hauler cannot

see the amount they collected all together, so probably the effect for them is less clear.

Second, schools with external hauler agreements have significantly higher confidence in their

measured values as external contractors charge schools based on weight recycled (82.2% vs.

63.0%).13

The tables also show that a significantly smaller proportion of self-haulers are in urban areas

(17.1%) than are schools with hauler agreements (27.7%). This is expected with urban schools

having better access the hauler companies services. The data show basically no difference in

other variables, such as per capita weight recycled, socio-economic status or region.

Table 5 in

13Shown inTable 13 in theAppendix.


21/67

21

Tables and figures compares School-to School participants to Community-to-School

participants based on their basic characteristics.

Clearly, schools that collect material from the community reported a significantly higher per

capita recycling weight (7.88 pounds) than their counterparts (4.63 pounds).

Table 14 in theAppendix shows that a higher proportion of schools that collect recyclables from

the public are in a suburban neighborhood (42.6%) than are schools that do not (26.2%). This

may be a result of the physical location of the schools and the community. Suburban areas might

be more closely intertwined with their surrounding communities, and this relationship reinforces

the positive experience associated with recycling (resulting in a significantly more positive

perception of the competitions impact on the schools recycling weight). Other variables, such

as such as socio-economic status or region, are not significantly different for the two contestant

groups.

Finally,Table 6 in


22/67

22

Tables and figures andTable 15 in theAppendixpresent a descriptive statistics breakdown by

whether or not a school perceived a positive impact of the competition on their recycling weights

(self-reported variable from the dataset of results).

Schools that perceive the competitions impactless positively are more likely to be self-haulers

(94.7% vs. 80.5%), have more confidence in their measurement (85.6% vs.74.8%), and are less

likely to be from the affluent neighborhoods. As it has also been shown, schools with no

recycling coordinator are significantly more likely to evaluate their own work more positively

and are more likely to have a coordinator.

There is another interesting difference between schools with positive and schools with less

positive perceptions: schools in suburban communities rate their own efforts significantly more

positively than other types of neighborhoods. This might also be a result of the fact that the

schools and their neighboring communities are closely intertwined; a school effort receives

constant positive feedback and support from the local community.

5.

Analysis

Question 1: Effect of Haulers and Self-haulers on Recycling Rates

Regression Analysis

This section attempts to test whether schools that self-haul recyclable materials present higher

rates of recycling weight per capita in comparison to schools that work with private hauler

companies. Furthermore, this analysis aims to evaluate how important is the hauling factor

compared to other school factors in shaping recycling rates among participating schools.

As described in the methodology section, the study performs a cross-sectional analysis of the

schools that registered and presented results in the Recycle-Bowl Competition of 2012. The

analysis provides results for the whole sample of schools that participated in the Recycle-Bowl

Competition, School-to-School subsample and Community-to-School subsample. The authors

decided to do this subgroup analysis because, as shown in the descriptive analysis, there are

significant differences between the schools that participate in each program.


23/67

23

Two different specifications have been used for each of sample. The first one is a bivariate

regression that compares the recycling weight per capita for schools that are self-haulers relative

to schools that operate with haulers. The second specification attempts to reduce the number of

omitted variables that could be influencing the recycling rates in schools by controlling for

factors such as: presence of a recycling coordinator, weighting method used by the school,

schools characteristics (type, zone, level), income, and region of the country.

Self-Hauler Effect on Recycling Rates

The self-hauler coefficient is the one that shows whether self-hauling is associated with a higher

recycling rate. Due to this, the authors focus their analysis during this segment on the first row ofTable

7in


24/67

24

Tables and figuresand the first row ofTable 16 in theAppendix.

Contrary to what was originally hypothesized, schools that self-haul do not present a significant

difference in the amount recycled compared to schools that work with a hauler. For the whole

sample analysis, the bivariate association suggests that schools that self-haul recycle more than

schools that have a hauler, but this difference in means (0.6 pounds per capita) is not statistically

significant. When more controls are added, the coefficient of interest flips sign, showing that

schools with haulers recycle more than schools that self-haul (0.5 pound difference). Again, this

difference is not statistically significant.

This pattern is reproduced in each of the subsample analysis. The bivariate analysis of the

School-to-School subsample suggests that schools that self-haul the materials recycle more than

schools that work with haulers. The difference in means between both groups is almost

imperceptible, 0.06 pounds per capita, and statistically insignificant. In the multivariate analysis,

the coefficient of interest presents a change in the direction of the relationship. In this case, and

holding all other variables constant, schools that self-haul recycle 0.4 pounds less than schools

with haulers. But again, the difference between groups is not statistically significant either.

Finally, the results for the Community-to-School subsample also present the same trend. The

bivariate analysis indicates that self-hauling schools recycle more than schools with haulers, a

difference of 2 pounds in the means per capita. But this difference is not statistically significant.

While, in the multivariate regression, holding else equal, self-hauler schools recycle an average

of 4.5 pounds less per capita than schools that have private hauler companies.

All in all, the different estimations imply that there is no real effect of self-hauling in the amount

recycled by the schools. This implies that self-hauler schools do not have a detectable advantage

in the competition over schools that operate with a hauler company.

Effect of Other Relevant Variables in the Recycling Rates

Although the hypothesis that states that self-hauling schools recycle more was not confirmed by

the evidence, the authors believe that some of the variables that are incorporated in the

multivariate regression analysis could be useful to identify key factors that drive recycling rates.


25/67

25

This segment of the analysis will look at the control variables used in the multivariate regression

in order to evaluate the effect of other factors in the recycling rates of the schools.

As the descriptive statistics have shown, the effect of recycling coordinators is not positive. The

result from the full sample suggests that, after controlling for different factors, having a

coordinator is associated with a decrease of 2.2 pounds per student recycled. This estimate is

statistically significant at the 1% level. Although the results are counterintuitive, schools that do

not have a coordinator could also be schools with an even stronger commitment to recycle. Like

the literature suggests, motivation is a key factor for determining the amount recycled by a

person. Another possible explanation is that schools that have coordinators recycle less even

before the competition, so this difference was there beforehand and is not due to the presence of

a coordinator; unfortunately there were not pre-data to test this hypothesis.

Still, the effect of the coordinator disappears when the analysis is conducted separately for each

type of competition within Recycle-Bowl. As can be observed, the coefficient for coordinator is

still negative both in the School-to-School and in the Community-to-School subsamples; but

those coefficients are no longer statistically significant. This implies that in the subsample

analyses, there is no real difference in the amount per capita recycled between schools that have

a recycling coordinator and schools that do not have one.

Table 16 in theAppendixpresents detailed results for the complete variable set. Another

important set of factors that could have some effect on the recycling rates are school

characteristics shown inTable 16. The multivariate model is controlling for three different

characteristics: school level, school type and zone/location of the school. In all different sample

analysis, the school type does not have any statistically significant effect on the recycling weight.

Although private and charter schools seem to recycle more than public schools, the difference

between groups is not statistically significant. In the School-to-School subsample, holding all

things constant, private or charter schools recycle around 1.6 pounds per capita more than public

schools. In the Community-to-School subsample, the gap in the recycling rates between private

schools and public schools is much important (8.4 pounds per capita). Still, none of these

coefficients are significant.


26/67

26

On the other hand, there are some significant differences in the school level variables. The full

sample result suggests that on average, and holding all other variables constant, high schools

recycle 2 pounds per capita less than elementary schools. This difference is statistically

significant at the 1% level. Middle schools and mix level schools have a tendency to recycle less

than elementary schools, but in those cases the difference is not significant. These patterns are

exactly the same in the School-to-School subsample, only that the magnitude of the difference

between high schools and elementary schools is slightly smaller (1.8 pounds per capita less for

high schools relative to elementary schools). In addition, for this subsample there also

statistically significant differences for between the amount per capita recycled by elementary

schools and mix level schools, where this former group seems to recycle more than the latter (1.6

pounds). The Community-to-School results show no effect of school level in recycling weights.

In addition, the location (or zone) of the school seems to be correlated with the recycling rates.

In the full sample analysis, holding all things constant, rural schools recycle 2.5 pounds more

than suburban schools. This is the only zone coefficient that presents statistically significant

differences, implying that the recycling rates of urban and mix zone schools are not different to

the rates of rural schools.

This factor also exhibits some differences in this subsample analysis. In the School-to-School

case, the magnitude of the coefficient is slightly smaller relative to the one presented for the fullsample. On average, and holding all things constant, suburban schools recycle 1.5 pounds per

capita less than rural schools. This seems reasonable because schools in the subsample have an

average per capita weight that is smaller than the one for the whole sample (4.6 and 5.13

respectively). Thus, the estimates for the subsample are expected to show smaller coefficients.

Another important difference with the full sample analysis is that in the School-to-School

subsample, urban schools present a difference in the amount recycled per student that is

statistically different from the rural schools. In this case, the estimation suggests that on average

and holding all other variables constant, urban schools recycle 1.6 pounds more per student than

rural schools. Likewise, the location of the school appears to have some effect in the

Community-to-School subsample. As in the results for the whole sample, holding other

variables constant, suburban schools present a recycling rate that is smaller than rural schools.

The magnitude of the coefficients for the Community-to-School subsample is around 6 pounds


27/67

27

larger than the estimate presented for the full sample, and similarly this subsample estimate is

statistically significant. In addition, this subsample analysis also indicates that schools in urban

areas have an average recycling rate that is smaller than rural schools. Holding all other

variables constant, urban schools that participate in the Community-to-School competition

recycle 6.5 pounds less than rural schools. This difference is statistically significant at

conventional levels. Finally as it was observed for the other two samples, there are no real

differences in recycling rates between rural and mix zone schools.

In the full sample results, the income proxy variable insinuates that, holding all other variables

constant, an increase of one percentage point in the number of students with free or reduced price

lunch in the school is associated with an increase in the recycling rates of 0.8 pounds per capita.

The subsample analyses also present this tendency of increasing recycling rates as the number of

low income students increases. Although these results seem to contradict the literature, which

suggests that high income families recycle more, these conclusions are not represented here, as

none of the coefficients are statistically significant at conventional levels.

Finally, the geographic region controls also present some differences in the analysis. Again, the

School-to-School subsample presents results that are quite consistent with the ones for the full

sample. In both cases, there are some statistically significant differences between schools in

different regions of the US. It appears that East North Central schools recycle more than schoolsin other regions. This is true for all the regions in the School-to-School analysis, showing a

difference that is between 5.5 and 13.2 pounds, depending on the region that is being compared

with. In the regressions for the full sample, the difference between the recycling rates of East

North Central and the ones of other regions varies from 1.4 and 8.5. However, four regions (East

South Central, the Middle Atlantic, Pacific and West-South Central) do not have a statistically

significant difference in the amount recycled per student compared to schools in East North

Central region. Lastly, the Community-to-School results indicate that there are not statistically

significant differences across regions.

Partial Conclusions

Haulers: In all the tables the authors find that there is no statistically significant

difference in the recycling rate between school with and without haulers.


28/67

28

Coordinators: In the whole sample, having a coordinator is associated with a decrease in

the amount recycled, but this is not the case for the subsample analysis. In both the

School-to-School category and in the Community-to-School category, the coefficient

becomes statistically insignificant when controlling for other factors.

School type: For all samples, there is no statistically significant difference between

public, private and charter schools recycling weights per capita.

School level: In the whole sample and in the School-to-School subsample it is observed

that holding other variables constant, high schools have an average recycling rate that is

lower than the estimate for elementary schools (around 2/1.8 pounds respectively). Mix

level schools also present some statistically significant differences in the recycling rates,

but only for the School-to-School subsample. From the Community-to-School analysis,

it can be concluded that there are no differences between groups.

School zone: In all samples the authors observe that, holding other variables constant,

suburban schools have an average recycling rate that is lower than the estimate for rural

schools. Both subsample analysis, there are statistically significant differences between

rural and urban schools. In the Community-to-School subsample, holding all other

variables constant, urban schools recycle 6.5 pounds per capita less than rural schools;

while in the School-to-School urban schools recycle 1.6 pounds per capita more than

rural schools.

Question 2: Effect of the Recycle-Bowl Competition

Up to this point the authors analyzed what are the factors believed to drive the recycling rates in

the schools that participate in the Recycle-Bowl Competition. It has been found that there are

some critical variables such as presence of a coordinator, schools level, and location of the

schools. Now, the authors test whether the recycling rates increase during the month of the

competition. Although this will not show a causal effect, it can be assumedholding all other


29/67

29

variables constantan increase in the recycling rates during the competition month could be

associated with the effect of the Recycle-Bowl Competition14

.

To analyze this effect, the authors first examined the differences in recycling rates before and

during the competition of schools that participated in the Recycle-Bowl, and the comparison

group schools that did not participate in the competition (SeeTable 8). The results show that

schools that participated in the Recycle-Bowl Competition recycle about 3 more pounds per

capita than non-participating schools during September (non-competition month) and

October/November (competition months). The table shows the difference in recycling weights

before and during competition months for participating and non-participating schools. The

results show that participating schools seem to have increased their recycling rates in 0.55

pounds per capita during the Recycle-Bowl Competition, although this difference is not

statistically significant. In the same way, non-participating schools seem to have decreased their

recycling rates in 0.02 pounds per capita during the Recycle-Bowl Competition compared to

September 2012, although this difference is not statistically significant. Thus, the difference of

those differences would imply that participating schools increased their recycling rates in 0.57

pounds per capita more than non-participating schools during the competition months, but this

result is not statistically significant.

These results would suggest that schools participating in the Recycle-Bowl Competition do notchange their normal recycling patterns during the competition. This conclusion can be read in

two different ways. One can interpret that schools exhibit some kind of recycling behavior that

occurs previous to the decision to participate in the program, and they self-select into the

competition. Another possible interpretation is that registration into the Recycle-Bowl

Competition is a factor that motivates the schools to recycle, and that effect can be seen from the

moment they registered instead of only during the competition month. In addition, since the

recycling rate increases for both participating and non-participating schools, there could also be

some seasonal or monthly effect that increases recycling rates for all schools during that period.

Unfortunately, the data used for this analysis do not allow the authors to say if either of these

interpretations is true. Additionally,Table 8 shows that overall schools that participated in the

14As we can assume that schools do not change dramatically between September and October/November, we are

confident that the main factor that changes is whether schools are in a competition month or non-competition month.


30/67

30

Recycle-Bowl Competition seem to have increased their recycling rates in 0.57 pounds per capita

more than non-participating schools during the competition months. Nevertheless, these results

are not only statistically insignificant, but they also only show simple differences and they do not

control for other factors, so regression analysis with schools fixed effects was performed.

As explained in the methodology section, the authors performed a difference-in difference

analysis that compares the recycling rates of schools in the Recycle-Bowl Competition with

schools that do not participate in the program (before and during the competition). As mentioned

before, the sample in this part of the analysis only consists of schools that have a recycling

coordinator. It is important to point out that the analysis only considers schools for which

recycling coordinators had pre-competition data. Thus, the regression is performed using panel

data of September, October and November for 441 schools.

Table 9presents the difference-in-difference regression analysis with school fixed effects

(explained by equation 3 in the methodology). This table indicates that, after controlling for

school characteristics that do not change over time (school fixed effects), schools that

participated in the Recycle-Bowl Competition increased their recycling weight by 0.44 pounds

per capita more during the competition than schools that did not participate (see interaction

coefficient inTable 9). This result is statistically significant at the 99% level. This is the closest

estimate of an impact of the Recycle-Bowl Competition in 2012. What this means is theRecycle-Bowl Competition had a positive effect on competing schools of increasing their

recycling rates by 0.44 pounds per capita during the competition, compared to non-participating

schools. Nevertheless, it is important to mention that this study cannot determine that this is a

causal relationship, and the authors suggest that the estimate be interpreted as a correlation

between the two variables.

Question 3: Analysis of data collection methods used by KAB

This section uses findings from the regression analyses on the first two research questions to give

recommendations to Keep America Beautiful on how they could improve the survey sent out to

schools during the competition. Improvements made to the survey will also enhance future

evaluations of the Recycle-Bowl Competition.


31/67

31

It is recommended that the survey be improved based on the following three principles:

It will have to collect all necessary data for further evaluations (including long-term

impact evaluations);

It will refrain from collecting unnecessary data in order to keep the workload of survey

responders at a reasonable level, and to increase response rates;

The survey has to meet all criteria for a good survey, such as phrasing questions that are

as short as possible, that have simple vocabulary and that are unbiased, it has to confine

to just one issue per question, and it has to avoid the possible bias carried by the structure

of a survey.

The authors recommend the following alterations to the current Recycle-Bowl registration sheet and

results report. These recommendations are a result of the evaluation of the program, and are intended

to assist Keep America Beautiful in future evaluations. A copy of the recommended forms is attached in


32/67

32

Tables and figures. The main suggestions are the following:

Recommend KAB to operationalize a system of school accounts for the registration

process, including the creation of a registration number per school and an automatic-fill

form for school information. Such a system would automatically fill in school

information the next time schools enter the system for either reporting results or

registering in a new competition year. This will assist KAB in tracking long-term data

per school to determine what changes or impacts the competition has on the schools

recycling rates. It will also allow KAB to more effectively track information per school

and to match the registration and results data for future analysis. This process would

simplify the registration and reporting process for schools as well, as they would have

their registration and reporting surveys semi-complete after entering the system.

Update questions and their order in the results report. The questions in this section have

been rearranged into an order that increases the flow between questions to assist

respondents in the survey. Questions have also been updated to meet the criteria of a

good survey and reduce bias.

6.

Limitations

The authors, per request of KAB, set out to do an impact evaluation of the Recycle-Bowl

Competition. However, upon receiving and analyzing the data, it was discovered that an impact

evaluation could not be completed. Therefore, this evaluation is an analysis of the association of

the competition with recycling behavior. There are three conditions for inferring causality: (1)

the cause must be statistically associated with the effect, (2) the cause must precede the effect in

time, and (3) spurious relationships must be ruled out (Adams, 2005). In this section, the

limitations of the conclusions of the first two research questions will be discussed. Since the

three conditions are not met with the available data, the conclusions only depict correlation

between variables and not a causal relationship. The third research question does not infer

causality, and thus it is not discussed in this section.


33/67

33

Since this study does not derive conclusions about schools that do not participate in the Recycle-

Bowl Competition, external validity concerns are not discussed. The schools that participate in

the Recycle-Bowl Competition are not representative of all schools within the United States or

any geographical or political subset thereof. Therefore, conclusions drawn from these analyses

only apply to the sample analyzed, and have no generalizability outside the sample.

Limitations Affecting Answers to Both Research Questions

The sample, regardless of the research question, may be affected by measurement error, which

could undermine the results. The competition is based on data that are self-measured and self-

reported via an electronic form. This carries the possibility of both erroneous measurements and

typographical errors. There are three ways this study tackles this issue.

First, outliers (schools with per capita recycling weights that are more than 3 standard deviations

from the mean) were excluded from the analysis. Second, there are two types of measurement

practices among schools: they either report actual weights or they report volume-to-weight

estimates. A dummy variable that the models control for reports that information. And third,

there is another set of dummies controlling for how much the schools trust their measurement

methods.

Despite all these, the authors cannot rule out the possibility of some effect of the measurementerror to key variables. Likewise, the authors cannot rule out the possibility that if there are

errors, they do not randomly affect certain schools, because some recycling coordinators may be

less careful in their reporting than others. Continuous improvements to data collection methods

KAB is carrying out will decrease the probability of measurement error. This study contributes

to better data collection methods by providing recommendations in the Question 3: Analysis of

data collection methods used by KABsection.

There is almost certainly some omitted variable biasin the estimations, limiting the internal

validity of the results. Complete data collection is almost impossible, and there are a number of

unobserved characteristics of these schools that could affect their recycling rates. Probably the

most important omitted variable the models do not control for is the existence of parallel

competitions adversely affecting the unique effect of Recycle-Bowl. There are a number of


34/67

34

other hard-to-measure characteristics that were omitted, such as motivation, competition-

awareness of the school population, personal characteristics of the coordinator / contact person,

proximity of facilities producing waste in community-to-school participants, income distribution

of students and teachers, racial mix of the school population, etc.

Major threats to statistical conclusion validity are heteroskedasticityand multicollinearity. The

studys calculations deal with those threats by performing multicollinearity testing with Variance

Inflating Factors and by presenting heteroskedasticity-robust standard errors in all the statistical

tables.

Limitations Affecting the Results of the First Research Question15

In order to answer the first research question, it would be important to make sure that the unique

effect of the hauler variablethat tells whether a school is a self-hauler or it has a contract with a

hauling company is measured. Here again, there is an issue ofselection bias, because for an

ideal analysis random assignment of the hauler variable would allow for statistical conclusions

with good internal validity. Of course, haulers are not randomly assigned to schools, and thus

the authors have to make sure to control for all observed variables that might be different

between the two groups other than the hauler variable and the key dependent variable (weight

per capita). This is done in the full model of the First QuestionCross-sectional Hauler

Analysissection ofthe methodology. (Also, seeTable 7). Unfortunately, due to a number of

omitted variables, the models do not control for unobserved characteristics, and hence suffer

from an omitted variable bias(see above).

Limitations Affecting the Results of the Second Research Question16

The second research question aims at determining a classical causal relationship. The second

criterion for inferring causality is clearly metthe Recycle-Bowl Competition precedes data

reporting. Whether or not the Recycle-Bowl is associated with higher recycling rate is examinedin the Question 2: Effect of the Recycle-Bowl Competition sectionof the analysis. Spurious

15Do schools that self-haul recycle more than those that work with hauling companies? How important is this factor

compared to other school factors in shaping recycling rates among participating schools?16Does the competition meet its objective of increasing recycling rates in participating schools? Is there also an

increase in recycling in the long term?


35/67

35

effects also need to be ruled out above the ones already mentioned, as to rule out the possibility

of alternative explanations.

With its first strategy, the analysis of the second research question compares pre-competition

data to data reported during competition for schools the authors managed to obtain pre-

competition data from. The schools that provided data were obviously not randomly selected

from the whole sample of participating schools, and thus the conclusions drawn for the analyzed

subset are not generalizable to the whole sample. Therefore, there is an issue ofselection bias.

The schools providing data could be tendentiously different from other schools. They all have

recycling coordinators, for example. Differences between the subset and the whole sample are

presented inTable 1 andTable 10,but above those there could be a number of unobserved

characteristics differentiating the subset from the whole sample.

In its second strategy, the study performs a difference-in-difference analysis between

participating and non-participating schools before and during the competition. The analysis uses

October and November monthly weights instead of the reported weights during the course of the

competition (Oct 15Nov 9). This should not affect internal validity, since the results of

monthly data analysis most probably underestimate the effect of the competition by diluting

competition data with non-competition data.

Ideally, in order to disentangle the unique effect of Recycle-Bowl, the likelihood of a school

participating in the competition should be exactly the same for the compared groups. This way,

random assignment would take care of any dissimilarities between the two groups that are not

attributed to the effects of the competition. The organization of the competition does not allow

for this ideal; therefore, there isselection bias. As it has been discussed, Keep America

Beautiful only has information about recycling weights for participating schools during the

competition. Ex-post efforts by the authors to collect data from before the competition and from

non-participating schools could not possibly have resulted in an unbiased comparison group.

Non-participating schools with pre-competition data from mostly come from the same state

(North Carolina), and even the same county (Wake County). This substantially affects the

internal validity of the results for question two. Since there were no data available for non-

participants except for their geographic location and the weight recycled, no comparison table


36/67

36

was prepared. There are a number of other unobserved variables affecting recycling that may be

very different for those two groups. However, by controlling for pre-competition recycling rates

and school fixed effects the model inherently controls for many characteristics driving those

rates and hence has better internal validity than the first strategy.

7.

Final Comments

The authors of this analysis of the Recycle-Bowl Competition sought out to show the effects of

the competition on recycling behavior. Results presented in this evaluation are not generalizable

to outside of the schools that participate in the Recycle-Bowl. The four central conclusions of

this analysis are:

There is no real effect in recycling behavior associated with having a hauler. This

determination comes from the lack of statistical significance in the regression analysis of

both variables.

There is no evidence that suggests schools in the competition recycle more during the

Recycle-Bowl than in other non-competition months. A lack of difference between

recycling rates previous the competition and during the competition could result from the

fact that schools self-select to participate in the competition and already partake in

recycling efforts.

The most important result of this analysis is that schools participating in the Recycle-

Bowl increase their recycling weights per capita by 0.44 pounds more than non-

participating schools during the competition. The 0.44 pound per capita difference

between participants and non-participants is seen as the effect of the competition on

recycling behavior.

Since the average recycling weight per capita in the School-to-School competition is

4.628 pounds, we can conclude that program participation is associated with an increase

of around 10% in the recycling rates.


37/67

37

The limitations to these conclusions are numerous and they decrease the internal validity of the

conclusions greatly. But, it is hoped that by outlaying all these limitations, the authors can

contribute to designing a data collection procedure that allows for future impact evaluations with

greater internal, and potentially external, validity.

Still, this analysis provides KAB with valuable data and results that they can use not only to

improve the implementation of the program and their fundraising efforts, but also to reach more

schools in the future.


38/67

38

8. Tables and figures

Table 1. Comparison between sample of schools with and without pre-competition data(only Recycle-Bowl participating schools)

Category/ variableTotal

schools

Schools with NO

pre-data

Schools with

pre-dataDifference

p-value of the t-test

of the diff.

Number of Schools 1118 848 270

% of schools 100% 75.85% 24.15%

Student and teacher

population

798.800 790.300 825.500-35.231 (0.398)

(596.7) (615.8) (532.4)

Average recycle weight per

capita

5.201 5.372 4.6640.708 (0.207)

(8.037) (8.424) (6.663)

Includes recycled material

from the public

0.176 0.221 0.03700.183*** (0.000)

(0.381) (0.415) (0.189)

Has a recycling coordinator0.673 0.588 0.937

-0.349*** (0.000)(0.469) (0.492) (0.243)

Self-haulers 0.131 0.171 0.004 0.167*** (0.000)(0.337) (0.377) (0.0609)

Note: Standard deviations in parentheses. * p


39/67

39

Table 3. Descriptive statistics by coordinator status(only Recycle-Bowl participating schools)


schools

Schools without

recycling

coordinator

Schools with

recycling

coordinator

Difference

p-value of

the t-test

of the diff.

Number of schools 1,118.00 366.00 752.00% of schools 100.00% 32.73% 67.26%

Student and teacher

population

798.800 794.800 800.700-5.900 0.877

(596.7) (593.8) (598.5)

Average recycle weight

per capita

5.201 6.765 4.4402.325*** 0.000

(8.037) (10.2) (6.615)

Includes recycled

material from the public

0.176 0.216 0.1570.059* 0.015

(0.381) (0.412) (0.364)

Has a recycling

coordinator

0.673 0.000 1.000-1.000 .

(0.469) (0) (0)

Positive perception of

the effect of RB on their

recycling weights

0.546 0.776 0.4340.342*** 0.000

(0.498) (0.418) (0.496)

Self-haulers0.131 0.361 0.019

0.342*** 0.000(0.337) (0.481) (0.135)



40/67

40

Table 5. Descriptive statistics by community involvement(only Recycle-Bowl participating schools)


schoolsSchool-to-school

Community-to-

schoolDifference

p-value of

the t-test

of the diff.

Number of schools 1,118.00 921.00 197.00% of schools 100.00% 82.38% 17.62%

Student and teacher

population

798.800 793.500 823.400-29.903 0.523

(596.7) (580.6) (668.1)

Average recycle weight

per capita

5.201 4.628 7.881-3.253*** 0.000

(8.037) (6.611) (12.42)

Includes recycled material

from the public

0.176 0 1-1.000 .

(0.381) (0) (0)

Has a recycling

coordinator

0.673 0.688 0.5990.089* 0.015

(0.469) (0.463) (0.491)

Positive perception of the

effect of RB on their

recycling weights

0.546 0.504 0.741-0.237*** 0.000

(0.498) (0.500) (0.439)

Self-haulers0.131 0.124 0.162

-0.039 0.144(0.337) (0.330) (0.370)



41/67

41

Table 7. Haulers and Non-haulers Regression Analysis Results(only Recycle-Bowl participating schools)

Full Sample School-to-School Community-to-School

Bivariate

Analysis

Full

Regression

Model

Bivariate

Analysis

Full

Regression

Model

Bivariate

Analysis

Full

Regression

ModelSelf-Hauler

Schools

0.631 -0.534 0.0605 -0.373 2.077 -4.475

(0.426) (0.555) (0.933) (0.650) (0.404) (0.336)

Coordinator-2.216** -0.318 -2.203

(0.005) (0.668) (0.527)

Type of weight

method

controls

X X X

School level

controlsX X X

Zone controls X X X

Income controls X X X

Region controls X X X

Constant4.869*** 10.34*** 4.574*** 12.48*** 6.439*** 7.919

(0.000) (0.000) (0.000) (0.000) (0.000) (0.088)

N 980 980 817 817 163 163

R2 0.001 0.138 0.000 0.225 0.006 0.350

Note: Public schools, Rural Schools and schools in East North Central subregion are taken as a reference for

comparison. P-values in parentheses. * p


42/67

42

Table 9. Difference-in-Difference Regression analysis with panel data of participating and

comparison schools (Effect of the program)

Category/Variable Coefficient

Recycle-Bowl Month

(Oct-Nov)

0.030

(0.718)

Interaction (participant

during competition months)

0.437**

(0.002)

School Fixed effects Yes

Constant19.789***

(0.00)

N 1282

R-sq 0.967

Note: P-values in parentheses. * p


43/67

43

Recommended Updates to Recycle-Bowl Competition Surveys

The following changes are recommendations based upon the evaluation presented in this study.

All changes are shown by the text in blue, except for the rearrangement of the questions for the

reporting results survey. The questions were rearranged to provide better flow for the schools in

answering the questions. Likewise, the honesty pledge for the reporting results survey has beenchanged to statements schools must agree with, rather than questions.

Registration Information Response

School Contact Information

First Name:

Last Name:

Physical Address:

City:

State:

Zip:

Email:

Phone Number:

General Information

1 The person completing this form is an authorized school representative such as a:

a Community Rec

Documents

Evaluation of The Recycle Bowl