Dialogue with marsha mueller

Exemplary Evaluations

Dialogue with Marsha Mueller

87

JODY L. FITZPATRICK

Editor:

Mueller:

Editor:

Mueller:

Marsha, the evaluation you and Betty Cooke conducted won the AEA Award for Exemplary Evalua-

tions in 1996. Obviously, there are many things which make an evaluation exemplary. What do you

see as the most exemplary aspect of your study? As I understand it, we received the award because of

the involvement of staff in the evaluation process. I like involving users in evaluations, but I get really

excited when users expect to be involved. This study is worth taking a look at because it shows what can

happen when motivated evaluation users work col- Jody Fitzpatrick

laboratively with evaluators. We created a demanding plan, given our budget. The

only way you could do what we did was to work as partners. Your use of staff in the evaluation was a primary piece our readers would be inter-

ested in learning about. So, let’s spend a little bit of time talking about that. Your use of on-site evaluators, people who also have program delivery responsibilities

at each site, draws on participatory evaluation and empowerment evaluation con- cepts. What do you see as the primary advantages of using program personnel as

on-site evaluators?

Our site evaluators were ECFE staff and members of Family Education Resources of Minnesota (FERM), the nonprofit organization created to support training and

evaluation of ECFE programs. Staff involvement was essential since one of the goals of the study was to build organizational capacity. Involving staff as evalua-

tors also has benefits for families, the program participants. Staff participation in the evaluation helps family educators think critically about families and about what they’re doing.

Another advantage is an economic one. Our grant was for only $150,000. We also had $20,000 from the Legislature. Without staff participation this project

would not have happened; it would have been impossible. Because staff did all the data collection we were able to make very strategic use of special help-the independent raters, for example, and nationally recognized experts, when we needed

them. We were able to maximize the use of the dollars we had available for the evaluation. The skill of staff in working with families was another advantage. We

needed data collectors who could establish rapport with potential study participants immediately-within a week after they enrolled in ECFE. In other words, we needed people who were used to interacting with families with young children. Finally, staff were a source of program intelligence. They helped us understand the intent of the program, its context and rationale, how staff make judgments about families and what they are trying to accomplish. Their insight

88 AMERICAN JOURNAL OF EVALUATION, 19(l), 1998

helped make the analysis relevant to the program. The program staff contributed heavily to design decisions.

Editor: We talk a lot today about building organizational capacity and thinking critically.

Tell me more about what these terms mean in your project.

Mueller: We wanted to do two things. First, we wanted to incrementally expand the pockets of evaluation expertise that are evolving in ECFE. Second, we wanted to enhance staff understanding of the families. We involved 50 ECFE staff; 28

served as site evaluators and 22 were members of FERM. Most are family educators, people who work directly with parents and young children. Yet, in their day-

to-day work, they do not always get to know families as well as they did during

the evaluation. In typical, day-to-day work with families, they usually work with groups of parents and children in an educational setting. During data collection,

they were with individual families in their homes; they had extended, focused

conversations with parents; and they took a very systematic look at those conversations when they analyzed transcripts. The data collection and analysis process

helped staff understand families in new ways. In looking systematically, staff viewed things differently. For example, they were often surprised at the range of skills and diversity of characteristics we found within our sample of low-income

families. For many staff, what we learned about the characteristics and skills of families challenged their mental models of what low-income families are like.

Staff were also intrigued by what we learned from examining different perspectives of change. A common observation of site evaluators following completion

of the final interview with parents was that the parents didn’t talk about all the

ways their parenting had changed, things the staff had observed when parents interacted with their child at the program site. Parent survey results, however,

showed that many more parents felt their behavior had changed than was apparent from staff or independent rater assessments. Recognition of different kinds of per-

spectives and discrepancies in findings provided opportunities for staff to discuss their practice in new ways. So, when I talk about evaluation as capacity building I mean thinking about and using the evaluation process as a learning opportunity,

a form of staff development or organizational investment, in tandem with conven- tional purposes.

Editor: Many evaluators, however, are concerned about making extensive use of program personnel to collect data. What concerns did you have? What steps did you take

to avoid problems?

Mueller: Of course we had concerns about data quality-how people go about collecting the data, how they handle it once they’ve got it, how they relate to study partici-

pants. I try to handle these concerns by understanding what people (FERM members and site evaluators in this case) are bringing to the table-their expectations for the evaluation, their skills or experience, and most importantly, their commitment to the task.

My job is to get to know an evaluation team very thoroughly. I’ve been involved in probably 50 evaluations, most of which used staff in one way or another. My first task is to get to know staff so I can plan appropriate training or other support into the evaluation process. In this case, we provided training at four intervals and allowed for a lot of practice up-front. We had a real technical mix of things staff needed to do. It was essential that they be comfortable with the tech-

Exemplary Evaluations 89

Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

Editor:

nologies they were going to use, the tape recorder, video camera, approaching

families and engaging them in the study as well as understanding the study and

evaluation logic.

We had a six-month pilot phase which was very important for learning and

fine-tuning skills-whether it was interviewing, managing the data, organizing

data files, taking care of transcripts, or whatever. I prepared two very detailed

handbooks which gave staff suggestions, down to specifics on coding, the struc-

ture for transcripts, timelines, etc. The pilot also gave me an idea, based on my

review of pilot data and conversations with site evaluators, about the quality of the

information we would get in the study. And that was really important. We had a

complex plan; we were taking some big risks. If staff could not meet quality stan-

dards or figure out how to implement the process in their district, I wanted to find

that out during the pilot. Betty and I also built in an auditing process. Each family

file (this included audio tapes, transcripts, survey responses, video tapes and site

evaluator process notes) was reviewed by at least two different people (another

site evaluator and a graduate research assistant). Betty and I also reviewed a sam-

ple of family files.

Who were the on-site evaluators? What other roles and responsibilities did they

have? What was their relationship to you? to their organization? How were they

selected?

Site evaluators were family educators and school district employees. We were

partners in this endeavor, neither Betty nor I was their supervisor. Site evaluators

were selected by their district coordinator. All had programming or administrative

responsibilities in addition to their participation in the evaluation.

Did you develop any guidelines for the district coordinator to use in selecting site

evaluators?

We asked the coordinator to select people who were interested and wanted to

learn more about their families and evaluation. We wanted people who were com-

mitted to the idea. Obviously, we also needed people who were willing to commit

the time to do it.

You mentioned that some of the site evaluators were administrators. Did they see

families?

Yes, ECFE is hierarchically flat; their people perform a variety of tasks and most administrators see families.

What was the educational background of the evaluators? And what were their

expectations of evaluation?

We had a mix of bachelor’s and master’s level people, even some Ph.D. folks. We

were able to do some sophisticated stuff; by that I mean having staff take care of

a full range of evaluation tasks and using multiple methods to collect data, not so

much due to their educational level as their interest and commitment to the eval-

uation. Some had been involved in prior ECFE evaluations. Some had not. It was

a purposeful mix of expertise. We wanted to expand the pockets of evaluation expertise that are evolving in ECFE, thoughtfully. In many cases more experienced site evaluators were paired with novices.

I noticed the site evaluators used the Stimulated Response Interviews (SRI’s) as an opportunity for teaching. Tell us about that.


Mueller: This is something we added based on staff experience in the pilot phase. The SRI

involves parents viewing segments of the videotapes and then responding to ques-

tions about what they observed. The questions posed to parents are very open-

ended. For example, “What’s going on here? What are your reactions to what you

saw?’ Parent responses are tape recorded and transcribed. The intent of the SRI is

to understand how parents talk about their behavior with their child when con-

fronted with it on a videotape. In the pilot, site evaluators realized there were a lot

of good things happening in the videotape that parents didn’t talk about, but the

evaluators couldn’t say anything about them without influencing parent

responses. As parent educators, however, they felt they were missing an opportu-

nity to reinforce good parenting skills. We decided to build in a teaching opportu-

nity. After the interview was completed, the site evaluators, if they wanted to,

could turn off the tape recorder and say, “There are a couple of things I really want

to show you.” This became a very powerful teaching tool and a piece of the eval-

uation methodology that districts are modifying and incorporating in their work.

When we analyzed information from parents about outcomes, study families

reported more changes in their children than did other low-income program fam-

ilies. Our hunch is that this was due to participation in the study, particularly the

learning aspect of the SRI which stressed observation and involved discussion of

parent behavior with the parent.

(Editor’s Note: We see here a phenomenon not unusual in evaluation and

research-that of changes in study participants being due, somewhat, to the study

itselJ i.e., the Hawthorne effect. What is noteworthy here is that the program

deliverers, seeing the beneficial effects of some components of the research,

incorporated these into their future programs. While the SRI was an additional

component that the study families experienced, they also received more individual

attention through home visits and interviews. Note Mueller’s observation that site

evaluators learned more because they usually dealt with families in groups on-

site; the study gave them an opportunity to make individual home visits.)

Editor: Let me move away from staff involvement for a minute and ask you about the

evaluation overall. What was the impetus for your study? Who initiated it and

why?

Mueller: This was the second outcome study of ECFE. The first study-a $20,000 grant

from the legislature to look at outcomes-made use of staff to interview parents

about outcomes. The sample for that study was mainly middle-income mothers.

The results were positive, but questions emerged: What can we learn about low-

income families in ECFE? What kinds of outcomes do low-income families dem-

onstrate? So, for the second study the focus was on what can we learn rather than responding to a legislative mandate. It was a discovery rather than an application

to a direct policy decision. At the request of FERM, Betty Cooke submitted a pro- posal to the McKnight Foundation for evaluation funds. McKnight awarded

FERM $150,000 for the study. The same year, the Minnesota legislature ear-

marked $10,000 per year for evaluation of ECFE.

(Editor’s Note: This use of evaluation for discovery rather than for a direct appli-

cation to a decision is one that has become more widely recognized in recent

years. Direct, instrumental use occurs and, as Cook (1997) has recently argued,


Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

may be more frequent than we have suspected. However, the use of evaluation for enlightenment is an accepted and valued purpose.) Who did you see as the primary audiences for information? The legislature?

I make a clear distinction between primary intended users and evaluation audi-

ences. The primary intended users of the evaluation information and process were ECFE staff, the approximately 2,000 family educators working in Minnesota school districts. They had questions about outcomes and wanted information

about outcomes to help them make decisions about how to improve their work

with families. Staff also wanted information about outcomes to share with legislators. The second purpose of the evaluation, as I mentioned before, was building

organizational capacity, so ECFE staff were also the primary intended users of the

evaluation process-certainly not the legislature, for they could care less about process use. Audiences for the evaluation, those who may need or want informa-

tion about the program and program outcomes, included legislators and other pro-

fessionals around the country involved in family support and education programs.

You note that a 60-member team participated in planning of all phases of the

study. How did that work? That’s a large group to process.

This team never met as an entire group. The nine group meetings held during the

30-month project included three planning sessions with FERM, five one- or two- day workshops for site evaluators and one meeting with FERM members and site

evaluators to review findings and formulate recommendations. Group meetings and workshops were carefully planned. Each meeting was designed to accomplish

evaluation tasks and to provide training and opportunities for staff to discuss what they were learning about families and evaluation. The questions for the study

were really established with the FERM group though there’s overlap between the

site evaluators and FERM.

So FERM members were the ones involved in the major planning sessions of the

study. Most of your interactions with on-site evaluators were in training for data collection though they were involved in a final meeting with FERM to review findings?

Keep in mind that almost all FERM members are ECFE staff and some FERM

members served as site evaluators. Evaluation purposes and questions and design decisions were established with FERM. We did some revision of our design based

on staff experiences with the pilot. Our training sessions, as I just mentioned, went well beyond skill development. Many FERM members (ECFE staff) participated

in the workshops even though they had no data collection responsibilities. Our review meeting was a day-long, very intensive and active working session. Staff discussed and debated findings and formulated their recommendations for inclu- sion in the report.

Let’s discuss your methodology briefly. You used a purposeful sampling strategy. Tell us about why you chose this strategy.

Random sampling did not fit our situation, and we also didn’t have the budget to implement random sampling effectively. We wanted to learn things about lower- income families so we purposefully involved districts that had higher numbers of lower-income participants. We also had to work around very practical constraints. We wanted to study families new to the program, and we wanted to involve them within a couple of days after they enrolled to limit the amount of exposure they


had to the program before we collected base line information. This is another example of how staffs program intelligence really helped. Let me add a side bar here. Apparently it’s common for parents to pick up “ECFE language” pretty quickly-parents pick up on how parent educators talk with them and how they

talk about children. So we had to move fast to get the base line data. We also wanted at least 100 families to complete the study and felt, given attrition, that we

should begin with 150. We wanted families with incomes of less than $30,000 but

fairly evenly distributed between three income sub-groups ($O-$9,999, $lO,OOO-

$19,999, and $20,000-$30,000). We also hoped our study families would mirror the ethnic diversity of low-income families in ECFE. To identify potential study

families we asked all new ECFE families enrolling during the first six weeks of the fall to complete a survey. Site evaluators recruited four to six of the earliest survey respondents meeting study group criteria to participate.

Editor: Why did you define “low income” as those with annual family incomes below $30,000? That’s pretty high. Median family incomes are close to that.

Mueller: A couple of points. We wanted to look at a range of low-income families. Part of it was practical. Income was broken down into six or seven categories on the

enrollment form; we wanted the three lowest. Also, I should point out that we’re looking at a specific program in a particular state. I think the 1990 Census state-

wide median family income was around $37,000. The 7-county metro area

median family income was around $41,000. When we look at study families, 72% had incomes less than $20,000.

Editor: What was your refusal rate?

Mueller: Very few families refused. Some site evaluators had to work harder than others. For the most part, staff skills in working with families really helped here. In some cases, they had to look at the first ten families instead of the first four to six that

met the selection criteria. For people who were reluctant to participate, the reason given was often not wanting someone to come to their home. So for some families, we did the interviews and videotaping at the center. We had a 79% comple-

tion rate. Families not completing the study typically dropped out because they moved or went back to work and were not participating in the program.

Editor: You used a mix of surveys, interviews, and videotapes of parents and children to collect information. How did you decide to select these particular methods?

Mueller: The strategies we selected depended on the evaluation questions themselves and our desire to understand what happens to families from three perspectives-parents, staff and independent raters. For example, the survey gave us descriptive information on families and their enrollment patterns. We used interviews because we learned in the earlier study that interviews provided in-depth information about how parents talk about their children and their behavior-they were very useful in helping staff understand what families know. Staff also learn a lot about families by interviewing them. We used observations with notes and the videotapes as another means of collecting information about behavior. I was skeptical about using video cameras initially. I felt it would be intrusive. Betty used videotaping in her dissertation research, but I wasn’t convinced. Before we made the decision I visited a program where videotaping is used extensively with teen parents. The videotaping was an accepted part of that program’s environment. It didn’t appear intrusive at all. As a matter of fact, it struck me as a great way to


help parents understand their own parent-child interaction. When we started the

study, very few districts were using videotaping or had video cameras so we spent time during the pilot learning how to use the technology and figuring out how and

where to borrow cameras.

Editor: What role did the FERM and others play in these decisions?

Mueller: FERM members were very involved in making decisions, and their reflections on past evaluations were helpful in shaping what we did. I would present different

options to the group and FERM members would chose what we did. Staff com-

ments and feedback from some academic reviewers also influenced how we struc- tured the analysis. The first outcome study had been criticized because staff

collected and analyzed data. Some academic reviewers for this study were

extremely skeptical about staff involvement in data collection. To address these

concerns, we built in the third perspective about parent behavior-independent ratings. We used Ph.D. candidates from the Institute of Child Development at the

University of Minnesota, recommended by one of the reviewers, to rate the videotapes. These independent raters were experts in analyzing videotapes of parent-

child interaction.

Editor: In regard to your results, you note that you found that correlations between the staffs’ ratings of parents’ behaviors and the independent raters’ ratings of video-

tapes were statistically significant (r = .33 for fall, .27 for spring, .23 for change scores). Were these correlations as high as you expected?

Mueller: We hadn’t done correlations between ECFE staff and independent raters before so

we had no idea of what to expect. I have a lot of confidence in staff assessments.

However, it was important to prove to ourselves that they’re doing a quality job so we wanted to compare them with independent ratings. The staff and independent raters were using different measures. The staff assessments were based on

their analysis of parent interviews, and the raters assessed videotapes of parent- child interaction, but they were measuring the same concept. It was of interest to us that the scores were in the same direction. The correlations were positive and significant. This suggests that although staff and independent raters used different scales, they rated parents’ behavior in similar ways.

Editor: Which of your results do you think show the strongest support for the program?

Mueller: I think what we learned points to both strengths and weaknesses of the program.

In terms of strengths, I would point to a mix of things rather than just one finding. First, low-income families voluntarily participate in this non-targeted program-

58% of new enrollees at study sites were low-income. Second, ECFE attracts a diverse group of low-income families in terms of prior knowledge of child devel-

opment and parenting skills, demographic characteristics, number of risk factors and social support. Third, parents are very satisfied with the program and report changes in their parenting skills and in their children. For example, 92% of parents reported positive differences in their awareness and understanding of children and child development, in their confidence as a parent, and in social support.

Seventy-two percent reported improvements in how they relate to their child. Changes in children’s behavior reported by parents included increased indepen- dence, improved language and communication skills, improved relationships with other children and more self-confidence. Finally, ECFE is helping parents learn more about child development and parenting-according to staff assessments,


Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

Editor:

Mueller:

34% demonstrated positive change in their knowledge and awareness of their own child.

(Editor’s Note: Low-income families certainly do participate in the program; however, sites that were more likely to have low-income families were selected in order to examine program effects on this group. So, the 58% low-income is probably not representative of all sites. And, some of those “low income” families have incomes quite close to the median.)

Let me ask you, which results do you feel are the weakest regarding program outcomes?

The piece that provides the most pause for thought is that knowledge change

exceeds behavior change. That’s an important piece, and the piece people talk about.

What’s your view on this?

We spent a lot of time talking about this, and staff recommended that districts

review all educational methods and look for ways to help parents observe, practice and reflect on parenting behavior and their parent-child relationships. They are acting on this recommendation.

What’s interesting about this finding are the different interpretations made by

different audiences. Legislators typically feel the percentage of families changing

their behavior (as assessed by staff or independent raters) should be much higher. Academics and professionals working in family support and education programs

are more apt to understand the tenacity of parenting behaviors and the difficulty

of changing patterns that are formed over many years. One of our reviewers, an expert in child development, predicted we would find limited, if any, change in parenting behavior after 6-8 months of program exposure.

Let me ask a little bit about how you analyzed the knowledge and behavior changes. You report most of your results in terms of percentage changes. How did you judge your results? That is, how did you determine whether the amount of

change was sufficient? Did you consider using statistical tests or setting explicit

standards? Let me take your last question first. No, we did not consider statistical tests of sig-

nificance to assess change findings. Given our design it would have been inappro-

priate to do so. How we analyzed knowledge and behavior change is briefly described in the summary and explained in detail in the full report. Look carefully at the statement of themes and findings. They are descriptive rather than evalua- tive, which is appropriate.

But, you do talk about change. You describe changes that occurred. The implica- tion is that these are meaningful. Did you set some criteria for the types of changes you expected? We were looking to see what would happen. The evaluation was for program staff and program improvement. It was sufficient in this case to look and be fairly descriptive about it. One of your issues of concern was the appropriateness of universal access programs for low-income families. You conclude that universal access is effective. On what basis did you draw that conclusion? Let me clarify the universal access statement. The theme actually reads “ECFE’s universal access approach is effective with many different low-income families.”


In essence, we’re saying this non-targeted approach is robust. One of our analysis

questions was, who benefits? We wanted to see if we could identify a set of char-

acteristics common to those parents demonstrating change. We couldn’t. Families that demonstrated change are diverse. Characteristics of parents we know something about accounted for very little of the variance in parent score change for

either staff or independent ratings.

Editor: So, you’re basing your conclusion on the effectiveness of universal access on the

fact that the demographic variables in the regression were not significant. But, the regression also showed that hours of participation was not significant. So, how

can you conclude it works?

Mueller: A couple of points on demographics and participation: As I note in the report, the

finding of no difference by demographic sub-group is not unique to this evaluation.

Other evaluations have found that although parents demonstrate change, outcomes could not be predicted by demographic attributes of participants. The participation finding is consistent with what we’ve learned about adults in other learning pro-

grams that are voluntary and flexible-people learn at different rates and have different reasons for coming, staying, or leaving. In this study, the distribution for

hours of exposure was quite flat. The average was 42 hours, but the range was 8 to 126 hours. This is not unusual in a program that is voluntary and flexible.

(Editor’s Note: The lack of effect for participation may be due more to the difi-

culty in changing parenting behavior with a relatively limited amount of contact,

e.g., two hours a week. As Mueller notes, one of their reviewers predicted limited,

if any change in behavior. The large range of participation would, if anything,

make it more easy to see an effect for participation.)

Editor: On a related issue, you note that for families with moderate and high level skills

the program maintained their skills. Is maintaining their skills sufficient? Should

the program focus only on families with low-level skills?

Mueller: On a simple level, it is helpful to know that a program doesn’t harm people;

nobody wants to fund, administer or participate in a program that demonstrates negative change. So, is maintaining skills sufficient? First let me explain what we mean by low, moderate and high. For staff assessments, low scores reflect devel-

opmentally inappropriate knowledge of child development or uncertainty about

what one does as a parent or uncertainty about their child’s behavior. Moderate

scores were assigned to parent responses demonstrating basic and appropriate

child development knowledge or parenting behavior. A high score was assigned when the parent response reflected a relatively sophisticated understanding of

child development and parenting skill. High scores reflect integration of develop- mentally appropriate knowledge and awareness of their child’s behavior in rela- tion to themselves and others. ECFEprograms work toward and support parent

development at the medium and high levels. In our analysis we were most concerned about the lowest group and whether positive movement was demonstrated.

OK, should the program focus only on families with low-skill levels? Segrega-

tion by skill level is not part of ECFE’s philosophy. One reason for using non-targeted approaches is to avoid the stigma attached to labels. ECFE’s philosophy includes the notion that all families with young children can benefit. A comer- stone of ECFE’s approach is based, in part, on the assumption that parent-child development is supported not only by staff (professional family educators) but by


other parents with young children. Parents coming with limited or inappropriate

child development knowledge and low levels of parenting behavior interact with parents exhibiting moderate or high skill levels. Parents not only hear from family

educators about appropriate practices but interact with and learn from their peers-parents who demonstrate those skills. Is this evaluation advocating

ECFE’s universal access approach over targeted strategies? No. What this evalu-

ation does do is provide some information about what happens to low-income

families in a non-targeted family support and education program. This is useful information. Most family support and early childhood programs are targeted to

low-income families. Universal access approaches are not typically considered; they are not an obvious choice.

Editor: Let me move to the dissemination of your results. One of the advantages of on-site evaluators concerns the ease in communicating results. How were your results communicated and used on-site? Did you train the evaluators in means for com-

municating results? Do you think they were more or less successful than an exter-

nal evaluator would have been?

Mueller: Let’s look at communication in two ways-dissemination externally and communication within ECFE related to application. I mentioned dissemination in my

summary. We started dissemination efforts during the pilot phase. With site evaluators and FERM members I made a conscious effort to provide them with differ-

ent ways of talking about the study every time we met. For example, I had a brief l-2 page update on the study that we talked about at each workshop. I also

included a two-page summary of the evaluation plan in each handbook. At every meeting we talked about what we were doing and why. Our workshop sessions were communication sessions. There was a lot of internal communication there.

But, we didn’t specifically talk about how to communicate on-site. They had a lot of materials and had to talk to other staff in their districts as well as state and local

decision-makers about what they were doing.

What’s most important I think, given the capacity-building purpose of the

study, is what staff are doing to share and apply what they learned with other dis-

tricts not in the study. I talked about some of that in my summary. Site evaluators took the initiative to plan and conduct workshops and are serving as consultants

to their colleagues (in other districts). There is a real emphasis on observation skills for parents and staff, pieces of the evaluation methodology are making their way into programming. Betty and I did not plan this and I’m not involved. It’s neat-it’s a great example of process use-I’m really excited about it. Would this kind of application happen if the evaluation was conducted entirely by external

evaluators? I doubt it, no.

Editor: Did you visit the sites?

Mueller: A few in the metro area. Our budget did not allow for extensive travel.

Editor: What prompted your visits?

Mueller: To some extent, curiosity. I like to have a sense for what’s going on. One visit occurred during the planning phase, and I wanted to observe how that site used

video cameras.

Editor: Could you describe to us the site which made the “best” use of the evaluation? How did they use it? What enabled them to be the “best” site in this regard?


Mueller: All sites were active. I honestly cannot tell you specifically who has used it more

than another. From what I’ve seen and heard, they’re making exceptional use of

the information.

Editor: How did you monitor what went on on-site?

Mueller: Our budget didn’t allow for on-site monitoring. What we did do was rely on the

workshop process to make sure staff members were clear about what needed doing and why, and our auditing process to monitor data quality.

We also had the site evaluators keep process notes-filling out their reaction to issues-how parents responded, what worked well, what didn’t. They kept track

of their emerging conclusions and their initial hunches. After they finished data

collection, we had them complete a survey which asked for their evaluation of the

process. I asked what they would recommend doing differently. They indicated they would all do it again, that they had learned more about the families and about

themselves. The down side they cited was the time factor. We pushed the limits of what could be done.

Editor: There were obviously many strengths to your study, but all evaluations have

weaknesses. If you were to do this evaluation again, what would you do differently?

Mueller: Oh my, there are always things you would like to change or fine tune. One weak- ness, which was brought out in the recommendations, ultimately comes back to

sampling. The recommendation noted that ECFE needs to learn more about how

they can better serve families of color. In the study, the ethnic mix was not what we expected. We needed a different way of engaging families of color in the study, particularly African American families. Also, there are questions we

wanted to get at but couldn’t. The most important of those was, “What do we know about child outcomes?’ We are trying some interesting approaches to child

assessment (Work Sampling System developed by Samuel Meisels from the Uni- versity of Michigan) in Minnesota’s Learning Readiness and First Grade Pre- paredness Programs, but there is no way we could have incorporated work

sampling into the evaluation plan since it is as labor intensive as the work we did.

With regard to the capacity-building purpose of this evaluation, it is interest-

ing to think about what could be done differently. This project required a major,

major investment of staff time. Site evaluators received a nominal honorarium, but it in no way covered their contributions. The evaluation process and findings

are being used and applied, the payback is pretty good-but I think there may be smarter ways (more efficient ways) to combine evaluation and capacity-building

purposes. You know, if ECFE got a million dollar grant for evaluation tomorrow I probably would not recommend that they do a million dollar study. I would probably suggest they consider multiple projects, involving more staff but in less complex projects. They have an extensive agenda (evaluation agenda) and the iterative, single project approach seems really slow.

Editor’s Commentary: The Minnesota Early Childhood Family Education program evaluation presents a nice contrast to the first evaluation reviewed in this series. This evaluation is formative in intent and actively involves staff in planning and data collection. Exemplary

elements include the active involvement of FERM members in the evaluation, the use of site

personnel for data collection and exploration of results, the development of and experimenta-


tion with innovative outcome measures (the SRI), and the obvious impact this evaluation, and

others in the series, have had on capacity-building in the organization. It appears that the

evaluation and the organizational environment have encouraged staff to contribute to a true

learning environment, looking at their processes and outcomes and experimenting with differ-

ent approaches.

While this evaluation has many noteworthy qualities, I wanted to comment on two con-

cerns. First, I am confused about the discussion of changes and outcomes in both our inter-

view and the report without reference to any standards, criteria, or statistical tests. This

study relies, instead, on percentages to report the observed changes, often in ways that sug-

gest rather large effects. Thus, Mueller notes that “the number of parents receiving low rat-

ings on the Parent Behavior Rating Scale decreased by 27%. ” This finding is also noted

prominently in the summary report. Further reading of the full report shows that the 33%

of parents who scored low on the pre-program measure was reduced to 25% on the post-

program measure. This is a 27% reduction; however, in all, only 8% of parents showed a

change in behavior from pre-program to post-program measures. This latter finding might

have been a more appropriate summary statistic to emphasize if one is relying on percent-

ages. Whether these changes are meaningful, in terms of program standards, or due to

more than chance cannot be determined. While the author discusses the purpose as

descriptive, I am concerned that many readers would not come to that conclusion when

reading about “outcome themes. ” While the author notes a different design would have

been required to use statistics, statistics and/or standards are frequently used in the litera-

ture to examine the import of observed changes. Randomized designs can be used to estab-

lish causality, but statistical signi$cance is frequently explored to identify relationships.

Further, the authors go on to use regression analyses and statistical significance of those

analyses to assess the universal access component. A combination of standards and tests of

statistical significance would have been useful to help readers judge the merits of the

described changes.

My second concern relates to the extent of interaction between the author and the site

evaluators. As Mueller states, they were certainly able to do much more in this study due to

the staffs involvement in data collection. And, I have no doubt that their involvement in the

data collection itself contributed to a change in the organization and their learning. Further,

the FERM members, some of whom were site evaluators, were actively involved in the plan-

ning of the study. These elements should be actively considered by other evaluators. However,

I was somewhat surprised at the author’s limited awareness of variation across sites and their

(possibly related) lack of opportunity to visit the sites, nine of which were in the metropolitan

area. There seemed to be a distance between the program and data collection, on the one

hand, and the administration, interpretation, and reporting of the evaluation, on the other,

that I had assumed was not typical of participatory evaluations. It is critical to observe the

programs we evaluate in action and, particularly when using on-site data collectors, to

observe staff engaged in data collection in order to have a fill understanding of both the pro-

grams of interest and the evaluation itself

Nevertheless, Mueller’s efforts have enlightened us and the child development commu-

nity. She and her colleagues have greatly enriched the organizational capacity of ECFE sites

through their work. In this way, the evaluation will have a far more long-lasting effect than

the results of this one study.

Exemplaty Evaluations 99

ACKNOWLEDGMENTS

This evaluation was supported by a grant from The McKnight Foundation to Family Resources of Minnesota (FERM) and funds appropriated by the Minnesota legislature.

I would like to commend the fifty ECFE professionals involved in the study: Dr. Betty

Cooke; Lois Engstrom with the Minnesota Department of Children, Families and Learning;

members of FERM, and the 28 district staff. This group demonstrated the important contribu-

tions users bring to the evaluation enterprise.

NOTE

1. For copies of the evaluation report or more information about ECFE, contact Betty Cooke,

Early Childhood and Family Initiatives Specialist or Lois Engstrom, Supervisor of Early Childhood and Family Initiatives, Minnesota Department of Children, Families and Learning, 992 Capitol

Square Building, 550 Cedar Street, St. Paul, MN 55101. Tel: (612) 296-8414, E-Mail: [email protected].

REFERENCES

Cook, T. D. (1997). Lessons learned in evaluation over the past 2.5 years. In E. Chelimsky & W. R. Shad- ish (Eds.), Evaluation for the 21st century. Thousand Oaks, CA: Sage.

Cooke, B. (1992). Changing times, changing families: Minnesota early childhoodfamily education parent outcome interview study. St. Paul, MN: Minnesota Department of Education.

Minnesota Department of Education (1986, March 1). Evaluation study of early childhood family education-Report to the legislature. St. Paul, MN: Minnesota Department of Education.

Mueller, M. R. (1996a). Immediate outcomes of lower income participants in Minnesota’s universal

access early childhood family education. St. Paul, MN: Department of Children, Families and Learning.

-. (1996b, August) Observations: Second year follow-upchanging times, changing families.

Phase II. Unpublished briefing report to Family Education Resources of Minnesota. Minneapolis. Minnesota.

. (1995). Changing times, changing families II-Spring evaluation guide. St. Paul, MN: Depart ment of Children, Families and Learning.

-. (1994). Changing times, changing families II-Fall evaluation guide. St. Paul, MN: Depart- ment of Children, Families and Learning.

Patton, M. Q. (1997). Utilization-focused evaluation, The new century text. Thousand Oaks, California: Sage.

Weiss, H. & Halpem R. (1990). Community-basedfamily support and education programs: Something old or something new? New York: National Center for Children in Poverty, Columbia University.

Documents

Dialogue with marsha mueller