METHODS FOR COMMUNITY MENTAL HEALTH RESEARCH

METHODS FOR COMMUNITY MENTAL HEALTH RESEARCH* JOHN C. GLIDEWELL, PH.D., IVAN N. MENSH, PH.D., HERBERT R. DOMKE,

M.D., MARGARET C.-L. GILDEA, M.D., AND A. D. BUCHMUELLER, M.S.W. St. Louis County Health Department, Clayton, Missouri

N THE St. Louis County Health Department we have been involved for I about two years in an attempt to plan and carry out a meaningful evaluation of the mental health program in operation in several communities in the county.

We wanted an evaluation which would be more nearly objective than a tabulation of judgments about improved mental health in the community. We were aware that this required the resolution of a number of problems which had plagued evaluation research for some years. In reviewing the kinds of problems which have been typical in such research, one might classify them as follows:

1. Criteria selection-the identification of the kinds of changes which a successful program should make in the people of the community.

2. Hypothesis formation-the statement of hypotheses about the expected changes, which are (a) conceptually clear and valid statements about improved mental health; (b) applicable to all the people of the population sampled; and (c) specific enough to permit valid and reliable measurement.

3. Assessment-the development and the empirical validation of efficient methods of measuring the expected changes in the people of the community. 4. Sampling-the selecting of a sample of communities which will permit

(a) as broad a generalization of the findings as possible; (b) a definition of the subsamples of people who respond to the program and those who do not respond; and (c) a definition of the population which the total sample represents, and of the subpopulations which the subsamples represent.

5. Design-the construction of a research design which will (a) separate the effects of the program from the effects of the many other influences operating in the community; (b) give results independent of the differing conditions under which the program functions in different communities; and (c) permit a definition of the conditions under which the program is successful, and those under which i t is not successful.

6 . Data treatment-the selection and application of statistical methods of maximum precision and best fit to the form of the design and the data.

7. Reporting-the communication of the findings in a manner appropriate * Presented at the 1955 Annual Meeting in a Round Table on “Research for the Evaluation of Com-

munity Mental Health Service,” Margaret C.-L. Gildea, M.D., Chairman. The research and service programs on which this paper is based are programs of the St. Louis County

Health Department. The investigation is supported by a research grant (M-592C) from the National Institute of Mental Health of the National Institutes of Health, Public Health Service.

38

JOHN C. GLIDEWELL E T AL.

both to the limitations of the research and to the guidance of the practitioner interested in program development.

I t is well known that an adequate solution to each of these problems- perfect solutions are out of reach-is a necessary condition for meaningful research findings. Accordingly, one must know how each problem is met in order to judge the promise of an evaluation research project. On the other hand, complete coverage is outside the practical limits of this paper, and we shall limit our discussion here to only two of the problems: 1) hypothesis formation, and 2) design. We plan to report our approaches and solutions to the other problems in the near future.

Because we think that, to a large extent, evaluation research must be custom-built to fit the service program being evaluated, we shall begin with a brief description of the community mental health program we proposed to evaluate.

The community mental health program in St. Louis County has four components, because a different approach has been developed for each of four degrees of disturbance in children. These four components of the program and the degrees of disturbance with which they deal are: 1) the Lay Education Services, designed to deal with the usual developmental problems of children; 2) the School-Centered Services, designed to deal with the more moderate degrees of disturbance which do not disrupt school attendance or family organization; 3) the services of the Child Guidance Clinic, designed to deal with disturbances of clinical severity which cannot be handled in the school setting; and 4) the services of residential treatment centers, designed to deal with problems so severe that a complete change of environment is required.

To describe the program in greater detail, let us turn to the first component, the Lay Education Services. This approach is intended to focus the attention of normal parents on the emotional problems they are likely to meet as their children grow up (2). The approach is called “Lay Education,” because it is conducted by lay discussion leaders for groups of parents. This service is carried out by the Mental Health Association of St. Louis, the organization responsible for its origin and development. The corps of lay discussion leaders are all volunteers. They are trained in discussion leadership techniques and theory in a series of workshops. The discussions center around a film or a play, using these aids as springboards for group discussions. The programs are offered in a series to parent groups in PTA’s, schools, churches, and elsewhere.

As indicated before, the Lay Education Services are aimed toward the parents of children with minimal troubles. This “Grade One” disturbance level includes children who develop symptoms in response to discernible environmental stresses, incidental to the developmental tasks of childhood.

39

40 METHODS FOR COMMUNITY MENTAL HEALTH RESEARCH

The symptoms are short-lived, and they do not disrupt school attendance or family relationships for any significant length of time. The demands on the family are not such that the parents seek psychiatric help. These problems, then, are such that they can be approached through the learning experiences to be found in free group discussions among parents.

The second component of the program-the School-Centered Services provided by the County Health Department to the schools of the county- is centered in the services of a mental health counselor assigned by the Health Department to the school. The counselor is a trained professional-either a psychiatric social worker or a clinical psychologist with special training in group therapy. The counselor assists the school personnel in case finding, screening of referrals, in-service training, and maintaining liaison with service agencies and clinics. A special and central function of the counselor is to conduct therapy groups for mothers ( 3 ) . The groups meet once a week in the school. They are composed of mothers identified in the screening process as those who must deal with problems in their children at the “Grade Two” level of severity.

This “Grade Two” severity level includes children whose disturbances are not so specifically related to precipitating factors as are those in “Grade One.” The disturbances are of longer duration and not so clearly episodic. Neither school attendance nor family life is seriously disrupted, but the behavior deviations usually cause concern on the part of the teacher and the mother. These, then, are problems amenable to resolution outside the clinic, and the technique of group therapy for mothers in the school has been developed to meet the need for the resolution of such problems (4, 5, 6).

The third component of the program is the familiar Child Guidance Clinic, which needs no elaboration. These teamwork services of the clinic are provided by the County Health Department for children with “Grade Three” disturbances-those whose symptoms are of long duration, disabling enough to threaten to disrupt the family, and actually disrupting school attendance. These children are often identified and referred as a part of the School- Centered Service, but they cannot be adequately dealt with within the school. They require the teamwork facilities of the Child Guidance Clinic.

The fourth component of the program includes the residential treatment centers, especially equipped for the care of severely disturbed children on a 24hour basis. In St. Louis County such centers are operated by several agencies. These centers treat children with “Grade Four” disturbances. Such children show illnesses severe enough to disrupt their school attendance, and they come from homes so disturbed as to block effective treatment while the child remains at home.

Looking back over the four components of the program-educational, school-centered, clinical, and residential-it is obvious that they require

~

JOHN C. GLIDEWELL ET AL. 41

increasingly richer resources in both budget and staff. Costs can be less than a dollar per year per child for the education services, and may reach $10,000 per year per child for the residential treatment centers.

People who have been interested in community mental health services for children have almost universally become aware of the difficulties of obtain- ing sufficient funds or sufficient staff to run the appropriate facilities. Accord- ingly, our research was initiated to demonstrate, if possible, the effectiveness of the first two, less expensive components of the program-the Lay Education Services and the School-Centered Services. An attempt is being made to determine whether these two services can demonstrably improve the emotional status of children or modify the attitudes of their parents. Further, if these services are effective they should reduce the pressure on the last two services, relieving some of the demand for staff and money.

More specifically, the research for evaluation was designed to compare the effects of the first two components a t three levels of operation: 1) a combined operation, in communities where both the Lay Education and the School-Centered Services are functioning; 2) a single operation, in communities where only the Lay Education Services are functioning; and 3) a control, where no community mental health service is functioning.

With this description of the fourfold program and the need for evaluation as a background, let us return to our two problems of designing a meaningful evaluation research. You will recall that we have said that, among other problems, we had to solve the problem of formulating conceptually clear and specific hypotheses about expected improvement in the mental health of the people in the community. In addition, we had to find a way to separate the effects of the program from the effects of the many other influences operating in the community-the problem of sampling and design.

HYPOTHESIS FORMATION If we are to make a meaningful evaluation of a community mental health

program, we must be specific about just what we will accept as evidence of the success or failure of the program. This means that we must formulate hypotheses stating just what kind of changes a successful program brings about in the people of the community.

The lack of specificity of hypotheses in research in the field of mental health has often been noted in the literature. The editors of the American Journal of Public Health, for example, have said: “A definite hypothesis, with a clearly stated plan for testing it, is still relatively rare in themental health field” (1).

In our work in the St. Louis County Health Department we found an abundance of suggestions for hypotheses. We found suggestions in the literature of medicine and the social sciences, we found them in the state-


ments of the objectives of the program to be evaluated, we found them in discussions of the experience of the service staff.

The suggestions included hypotheses about improved mental health in terms of changes in such things as creativity, role flexibility, anxiety, ego functions, narcissism, compulsiveness, guilt, displacement, projection-all of which hypotheses are conceptually related to mental health, but difficult to test empirically. Other suggestions involved changes in school achievement, number and intensity of interpersonal relationships, number of demands made in interpersonal relationships, frequency of loss of emotional control-all testable hypotheses but not always so clearly related to improved mental health.

An example of one possible hypothesis will serve to illustrate and clarify the difficulties we encountered. Let us say that a successful community mental health program will bring about increases in the number of interpersonal relationships established by children. The hypothesis has specificity. I t deals with something that one can count with reasonable accuracy. I t deals with something that can be observed directly a t home and a t school.

The hypothesis has, however, an obvious limitation. An increase in the number of interpersonal relationships established by a child may mean for one child that his mental health has improved. It may mean for another child that his mental health has deteriorated. The hypothesis about increased number of interpersonal relationships, then, would apply to some children, but certainly not to a majority of children.

There is also a second difficulty. Whether or not such an increase means improvement will depend on many things. I t will depend on how many interpersonal relationships the child has established before the increase began, the intensity of these interpersonal relationships, and the emotiona1 tone of the relationships. In short it will depend on the total personality conjigura- tion ofthe child before the change began.

This illustrates two general difficulties with very specific hypotheses: 1) The more specific we make a hypothesis, the fewer are the children to whom it applies. 2) The more specific we make a hypothesis, the less it tells us about improvement, because we can judge improvement only within the context of the personality of the whole child before the change began.

To try to resolve these two difficulties we turned to another approach. We asked our service staff for descriptions of individual children-a picture of a whole, living child. In fact, we asked for two descriptions of each child- one before contact with the program, and one after such contact. We also asked the staff to try to avoid psychiatric terminology in the descriptions.

Following are abstracts of two examples of the “before and after” descriptions we obtained:


Berth before contact. Bertha was an aggressive, demanding, and flighty child. She kept her third-grade classroom in an uproar. She was always in and out of her seat, always angry a t some child, and often in fights. Her intensely demanding relationships spread among many children. She seemed to have poor control over her feelings, was easily distracted, seldom finished a task, and was failing in school.

Berth after contact. After a year, Bertha’s demandingness moderated. Her relationships did not involve so many other children. She showed a more reasonable emotional control, and was less flighty. While not too persistent, she could usually stick with a task until she finished it. Her schoolwork improved.

Amy before contact. Amy was a timid child. She had one very close friend in her third- grade classroom, and seldom spoke to anyone else, except the teacher. She was willing to join a game if urged but became very ill-at-ease in any group. When alone, however, she approached most tasks without tension, and worked steadily. She was disturbed only when she was forced to leave an unfinished task. I t was only a t such times, in fact, that she showed any resistance to demands made by others.

Amy after contact. After a year, Amy was more ready to interact with other children, had several friends in the classroom, and entered games with great personal involvement. She occasionally became quite resistant when other children wanted to change games. Otherwise, she made only moderate demands on her playmates. She still showed great persistence in her schoolwork, and insisted on completing one task before beginning another.

It is readily apparent that the two children in these descriptions changed, generally, in opposite directions. I t is also apparent that they changed specifically in different ways, or along different “dimensions” of behavior. If we ignore, for the time being, the directions of the changes in these children, we can see that the “before and after” descriptions of the two children con- tain several common dimensions. Bertha at first made many demands on others; Amy made few. Later Bertha’s demands decreased; Amy’s increased. But all these descriptions fell along the dimensions of demandingness. Bertha’s emotional control became stronger, Amy’s remained fairly rigid, but again the dimension was the same-degree of emotional control.

If we carry this type of analysis further, we can find a third and a fourth dimension along which both children are described-the number of interpersonal relationships established, and the intensity of these interpersonal relationships.

At this point in the analysis it becomes apparent that we need to find a way to deal with all four of the common dimensions a t the same time and in a single integrated unit. We need to do this because, to solve our problems, we must judge improvement in terms of the totul change from before to after, and at the same time, place the specific change within the context of the total personality configuration before the change began.

As a first step in accomplishing this integration, we may, for simplicity, assume that we can identify three points along each dimension-the two extremes, and a middle or moderate point. Thus, a child may make many


SOME DIMENSIONS OF CHANGE IN CHILD BEHAVIOR

*Lse =Loose emotional control. Flx=Flexible emotional control. Rig=Rigid emotional control.

FIG. 1

demands on others, a moderate number of demands, or few demands. With respect to emotional control, we can identify a very loose control a t one extreme, a flexible control in the middle, and a very rigid control a t the other extreme.

T o complete the integration of our example of four dimensions with three points on each, we can use a chart such as the one shown in Figure 1. This chart contains 81 cells. Each of these cells represents a type or class of children. Each such type or class is different from the other 80 with respect to these 4 dimensions of child behavior.

To make use of the chart to solve our problem, let us return to Bertha and Amy. Before contact with the mental health program, Bertha made many demands on others, showed a loose emotional control, maintained many intense interpersonal relationships. We can now locate a cell in the chart which describes Bertha a t that time. The symbol B1 has been entered in that cell. Now, after contact with the program, Bertha’s demands moderated, her control tightened, her many interpersonal relationships decreased, but their intensity did not. We can similarly locate the cell which describes Bertha after contact with the program. The symbol Bz has been entered in this cell.

If we proceed in the same manner we can locate Amy, before and after contact with the program, in cells A1 and A,, respectively.

Having located our two sample children on the chart, we can now describe

JOHN C. GLIDEWELL ET AL.

the change in the two children in terms of the change from one cell of the chart to another. I t , i s important to notice that when we describe these changes in terms of the movement from one cell of this chart to another, we are dealing with four dimensions of change at the same time. Thus, you can see that we have created an integrated unit, describing the behavior of a child along four dimensions a t once. These changes are indicated by the arrows on the chart.

Now, let us see whether this approach resolves the problems we originally encountered. You will recall that we found that the more specific a hypothesis about desirable change, the fewer were the children to whom it applied.

Any change in one direction along one dimension on this chart may represent improvement for only a few children. This approach, nevertheless, resolves this problem, because improvement is not dejned as change in any one direction along any one dimension. Improvement is defined as movement from one cell of the chart to another. Such movement may be along any dimension and in any direction, and applies to only those children who start in a given cell.

Turning to the second difficulty with the single-dimensional specific hypothesis, we found that judging whether any particular change represented improvement was dependent upon the total personality configuration of the child before the change began. The approach to the problem, which is represented by our chart, begins with determining where the child is before the change begins-location of the cell in which the child falls before the change. Knowing this, we may then select from the remaining cells those to which movement would represent improved mental health and those to which movement would represent deteriorated mental health. With this approach our measure of improvement automatically takes account of the point where the child was before the change began.

As has been indicated, improvement or deterioration in mental health is defined in terms of movement from one cell of the chart to another. This means that we need a method for determining whether any given movement represents improvement or deterioration-in effect a calibration of the possible changes. There are any number of methods for such a calibration. In our study we attempted to define, a t the conceptual level, each of the dimensions in such a manner that the middle point represented the optimal mental health. This would mean, if our concepts were valid, that improvement is defined as movement toward the middle cell of the chart. The first step in validating this calibration was to ask our service staff and consultants to respond to an especially constructed questionnaire. This questionnaire contained descriptions of children who fell into each cell of the chart. Each respondent was asked to indicate in what respect each child would improve if the program were successful. The results clearly confirmed that, using the

45


particular dimensions as we had defined them, improvement was conceived by the staff as movement toward the middle. These judgmental results were then tested empirically in a series of pilot studies. In these validity tests our definition of improvement was compared to that determined by independent diagnoses, based on individual studies by a psychiatric team. By these methods we can have reasonable confidence that the scale of over-all improvement-in this case defined as movement toward the middle cell- has clinical as well as empirical meaning.

Up to this point we have talked in terms of four dimensions of behavioral change in children. One may question whether, with only four dimensions, we have really taken account of the total personality configuration of the child. It seemed necessary to confine ourselves to four dimensions in this discussion because even four seemed to make for a pretty complicated ex- planation. The fact was that, in our work in St. Louis County, we found we needed to work with seven dimensions in order to cover all the changes which our service staff had identified in their past experience.

The dimensions selected were those which had meaning in terms of the particular objectives of the St. Louis County program and in terms of findings previously reported in the literature. As a matter of interest, but not necessarily as recommendations for research on other programs, the dimensions were: 1) number of interpersonal relationships established, 2) intensity of interpersonal relationships, 3) demandingness in interpersonal interaction, 4) emotional tone of interpersonal interaction, 5 ) favored objects of interpersonal relationships-adults or children, 6) extent of emotional control, 7) achievement drive in school tasks.

When we consider the complexities of human personality, 7 behavioral dimensions may seem pitifully few. In view of this it seems important to remind ourselves that our hypotheses are not formulated in terms of only 7 kinds of behavioral change. The kinds of change which one can describe with 7 dimensions are surprisingly large. To illustrate, you will recall that with only 4 dimensions we could describe 81 different classes of personality configurations, and over 6,000 different kinds of changes from one class to another. The number of classes-or cells-in such a chart as we have used here increases much faster than the number of dimensions. With our 7 dimensions we can describe 2,187 different classes of personality configurations. When we begin to identify the different possible changes from one cell to another, the possibilities are, for all practical purposes, infinite. We anticipate no difficulties with a sample of 1,000 children.

In review, we have tried to demonstrate that we must be specific about the kind of changes we are willing to accept’as evidence of improved mental health of the people in a community. We have also tried to demonstrate that we find ourselves in an untenable position if we interpret specificity


to mean that we must deal with specific dimensions of change, one a t a time. The more specific our hypotheses about the change, the fewer the people to whom it will apply, and the less meaning it will have as evidence of improvement.

On the other hand, we can collect objective data by taking measures along a relatively few specific dimensions, and then define improvement in terms of changes along all these dimensions a t the same time and within the context of the total personality configuration of the person before the change began. We have proposed that this can be done, and we have tried to demonstrate a method for doing it.

RESEARCH DESIGN AND CONTROLS Having formulated a set of hypotheses about the effects of the program,

we turn to the problem of designing a reasonably precise test of these hypotheses. The basic problem in our work was to compare the effects of the three levels of operation of the first and second components of the community mental health program in St. Louis County. You will recall that these levels of operation were: 1) a dual or combined operation of both the Lay Education and the School-Centered Services, which for brevity we will now call the “Dual” program; 2) a single operation, including only the Lay Education Services, which we will now call simply “Lay Education”; and 3) a control, involving no community mental health program.

To begin, let us review the field conditions which set limits on the sampling and design possibilities. First, there is available a sample of approxi- mately 1,000 families, each having a t least one child in one of two third- grade classrooms in each of 15 schools in St. Louis County. Because we want to deal with three levels of programs, we need to organize this sample into groups of three experimental units, each group being as homogeneous as possible. An attempt to arrange schools into homogeneous groups of three was a failure even for schools in the same geographical areas. The next best bet for a workable experimental unit was the classroom. The pupils from the areas served by a single school could be assigned a t random to the two third-grade classrooms, giving a desirable homogeneity for the two classrooms. Because there were only two classrooms to the school, however, it became necessary to create a design which would permit us to assign the three levels of the program only two a t a time. These limits led to the choice of an incomplete blocks design. Such designs permit the assignment of three levels of a treatment two a t a time, in our case, in the following manner.

There are three possible combinations of the three levels taken two a t a time: Dual-Lay Education, Dual-Control, and Lay Education-Con- trol. Each of these three combinations of program levels can be assigned to a school, One of the two classrooms receives one level; the other receives the


THE BASIC DESIGN

TI pi-l pq

El El [Zq

Lay Educ Lay Educ Lay Educ

Control Control Control

FIG. 2

second level. If this method of assigning levels of programs is followed through nine schools in three groups of three, the result is the arrangement shown in Figure 2. In our work this arrangement constitutes the basic design.

At this point we shall turn to another community reality which affected the design. Three of the 15 schools available had already completed three years of experience with the Dual program. Administratively, only minor modifications in previous services were possible. This made the combination “Dual-Lay Education” the only appropriate assignment. To deal with this factor, 3 additional schools with this combination of program levels were added to the design, yielding the arrangement shown in Figure 3.

Now let us turn to a further complication. You will recall that in the design the control classrooms occur in schools in which either the Dual program or the Lay Education program is functioning. Under such conditions, intraschools or intracommunity communication might have some effect on the control classrooms. To test this possibility, three control schools were added to the design. In these schools no program is functioning, and both

THE BASIC DESIGN AND THE FIRST ADDITION

Fourth r] PI] First Addition Year Schools Lay Educ Lay Educ Lay Educ

First

First

First Lay Educ

FIG. 3

Lay Educ

Basic Design

JOHN C. GLIDEWELL ET AL.

THE TOTAL DESIGN WITH THE TWO ADDITIONS

49

Fourth r] rl mkirst Addition Year Schools Lay Educ Lay Educ Lay Educ

Lay Educ Lay Educ

First Year Schools Control Control Control

First Year Schools

Basic Design

FIG. 4

third-grade classrooms are control classrooms. The 15-school design, with this second addition, is shown in Figure 4.

At this point we may move to consider some of the conditions which might affect the success of the program. First, there are nine schools having a Dual program and requiring the services of a psychiatric social worker in the school. There is the possibility, of course, that any effects we find in one school may be due to the special skill of the worker in that school. T o meet this problem, it was necessary to assign the available sample of three workers in a systematic manner, so that each of the workers would contribute equally to each of the program levels. A latin square arrangement met these require- ments and the workers were assigned as shown in Figure 5.

Continuing the problem of the control of the conditions under which the program functions, we must face the problem of the assignment of the lay leaders who conduct the group discussions in the Lay Education program. It was possible to study the leadership style of a sample of 16 lay discussion

THE ASSIGNMENT OF WORKERS TO THE NINE SCHOOLS WITH THE DUAL PROGRAM

Dual- Worker R Worker S Worker T Lay Education

Dual- Worker S Worker T Worker R Lay Education

Dual- Worker T Worker R Worker S Control

FIG. 5


THE ASSIGNMENT OF THE WORKERS AND LAY LEADERS

Dual- Worker R Worker S Worker T Lay Education Leader X Leader Y Leader Z

Dual- Worker S Worker T Worker R Lay Education Leader X Leader Y Leader Z

Dual- Control

Worker T Worker R Worker S Leader X Leader Y Leader Z

Lay Education- No Worker No Worker No Worker Control Leader X Leader Y Leader Z

FIG. G

leaders. This study made it possible to form 3 teams of leaders so that the leadership style is homogeneous within the teams and heterogeneous between the teams. Each of the 3 teams was then assigned to one of the 3 replications or columns of schools. Under this arrangement, each team contributes equally to each level of the program and each team is paired with all of the workers, as is shown in Figure 6.

When we continue still further the consideration of the conditions under which the program operates, we find that we expect different degrees of success with children of different levels of severity of disturbance. You will recall that the two component services to be evaluated are aimed at dealing with disturbances of severity Grades One and Two. Our sample, however, is a population sample, including all grades of disturbance. Accordingly, it was necessary to divide each classroom into subclasses according to the severity of disturbance found in the children before the program began. By this method the data can be treated separately for children in each grade of disturbance. This permits not only the control of the severity of disturbance but also a test of the hypothesis that the effects differ in samples of children having different degrees of disturbance.

As may be apparent, the design as described up to this point does not provide for the control of (a) differences in socioeconomic status of families, (b) differences in social organization of the families, and (c) differences in the degree of direct contact of the mother with the program or with other mental health resources. These factors cannot be controlled in advance; they must be taken as we find them in the sample. Accordingly, some statistical controls are required.

To the degree that the two classrooms within a school are homogeneous with respect to the three factors not amenable to prior control, the incomplete blocks design will largely eliminate the contribution of these differences to the error of the experiment, and our problem is met. At the same time, all three factors are being measured a t the initiation of the research. In this


way we can test the within-school homogeneity. If we fail to find such homogeneity, the three factors will be handled by covariance analysis, leaving the effects of the three program levels independent of these conditions.

SUMMARY 1. We have presented a brief review of a community mental health pro-

gram having four components-education services, school-centered services, clinical services, and residential treatment centers. The successive components are equipped to deal with disturbances of increasing severity in children, and each component makes increasing demands on the community resources of the staff and funds. Accordingly, we have maintained that it is essential to determine the effectiveness of the two less intensive and expensive services.

2. We have presented a possible approach to hypothesis formation which would permit the definition of improved mental health (a) in terms of the changes along many dimensions a t the same time, and (b) within the context of the total personality configuration of the child before the changes began.

3. We have presented an example of an evaluation research design which will permit a test of program success which is relatively independent of (a) the conditions under which the program functions, and (b) the effects of other influences operating simultaneously in the community.

Our work has led us to believe that the conduct of research to provide a meaningful evaluation of community mental health programs is possible, in spite of the difficulties involved. We hope that our two examples of methods of meeting some of the difficulties support this belief in the feasibility of meaningful evaluation.

REFERENCES 1. AMERICAN PUBLIC HEALTH ASSOCIATION. The Fifth International Congress on Mental

Health. Am. J. Publ. Health, 44: 1363, 1954. 2. BRASHEAR, ELLEN L., ELEANOR T. KENNEY, A. D. BUCHMUELLER, and MARGARET

C.-L. GILDEA. A Community Program o f Mental Health Education Using Group Discussion Methods. Am. J. Orthopsychiatry, 24: 554, 1954.

3. BUCHMUELLER, A. D., and MARGARET C.G. GILDEA. A Group Therapy Project with Parents of Behavior Problem Children in Public Schools. Am. J. Psychiatry, 106: 46, 1949.

4. - . Group Therapy for Parents o f Behavior Problem Children in Public Schools: Report o f Work in Schools in St. Louis County. Paper read before the First Annual Meeting of the International Congress of Group Therapy, Toronto, Canada, August 1954.

5. BUCHMUELLER, A. D., FRANCES PORTER, and MARGARET C.-L. GILDEA. A Group Therapy Project with Parents o f Behavior Problem Children in Public Schools: A Comparative Study of Behavior Problems in Two School Districts. Nerv. Child, 10: 415, 1954.

6. KAHN, JANE, A. D. BUCHMUELLER, and MARGARET C.-L. GILDEA. Group Therapyfor Parents o f Behavior Problem Children in P~bl ic Schools: Failure o f the Method in a Negro School. Am. J. Psychiatry, 108: 351, 1951.


DISCUSSION SIBYLLE K. ESCALONA, PH.D.:* Formal research to evaluate the effects

of mental health programs belongs in the category of enterprises which are exceedingly easy to criticize and impossibly difficult to do. I should like to start out with the flat assertion that as yet no one has developed an unex- ceptionable research design in this area even on paper-much less has anyone been able to carry out such a study and been able to avoid the familiar pitfalls of circularity, contamination and the handling of variables known to operate but remaining uncontrolled.

When confronted with an impossible task in research or any other context I know of, there are only two reasonable courses of action. One is to abandon the attempt; the other is to keep on working but to direct one’s effort toward a very clear realization of what it is one cannot do, and why. The authors of this paper chose the second alternative and deserve our gratitude for their courage, among other things, in wrestling with so formidable a task.

In relation to the St. Louis study the only contribution I can make is to point to some of the aspects where because of technical and theoretical difficulties the procedures being carried out must in my judgment fall short of the stated aims. The authors emphasize the desirability of finding measur- able changes in the behavior or characteristics of the population under study which will be unequivocally related to mental health. As you know, the team chose to develop systematic forms of expression (in terms of dimensions) for what might be called a combination of the clinical experience and the theoretical convictions of the staff. They asked their workers: What do you see in your clients that reflects a deviation from a state of mental health and what are the changes in people which you have come to recognize as being associated with an improvement in mental health?

From the point of view of research strategy, and particularly in the interest of making these kinds of judgment amenable to statistical treatment, I believe that this method can be a significant advance. It is desirable to realize, however, that neither the ordering of such material on continua or along dimensions, nor numerical ratings, can alter the essentially judgmental, self-contained and partially subjective nature of the raw data! From a strictly logical point of view the following sources of error seem important. Mental health workers specify certain indices for the presence of a degree of malad- justment, for instance the number and quality of interpersonal relationships maintained by a child. Theoretically it is possible that they do so because their training has led them to regard this as a focal point, or because it is an aspect of life which is accessible to their observation, or because the particular group of children who, through a highly selective process, end up as

* Professor of Psychology, Albert Einstein College of Medicine, Yeshiva University, New York, N. Y.

DISCUSSION: SIBYLLE K. ESCALONA 53

participating in a preventive or therapeutic program happen to manifest disturbance and improvement very clearly in this particular realm. In other words i t is entirely possible that these dimensions represent the mental health worker’s perception, and his working theory, more than they represent the actual phenomena. If we really knew the range of behaviors or characteristics which, under known situations, represent healthy and adap- tive personality functioning, perhaps the need for programs of this kind and for their evaluation would have become much less acute.

A similar source of possible error comes in a t the next step, namely, the field worker’s report on changes that do take place. It is in the nature of the beast that human beings change continually, that such change is reflected in an infinite variety of ways from the quality of their feeling and thought processes to each and every action they may perform. Only a small fragment of the changes actually taking place becomes observable to a teacher, or clinician or classroom observer, or to the psychologist administering the test. Again it is impossible to know whether or not the fragment of behavior selectively observed is representative of the whole and whether the observed behavior changes really refer to improvement in mental health or are related to situational factors or any number of other things.

One could easily go on and enumerate other uncertainties which are actually built into the research design, albeit largely by necessity. One of the conclusions at which I a t least would arrive after such an analysis is that, strictly speaking, certain changes can be registered or reflected by this method, but the term measurement is a euphemism, as the measuring in- strument is not the written-down formulation of dimensions but an aggre- gate of human beings performing judgments, who had not been subjected to tests of reliability, and their judgments are not capable of being validated in the technical meaning of the term.

When it comes to the crucial problem of the degree to which observed changes are due to the absence or presence of certain types of programs the authors have themselves mentioned several sources of error which must re- main uncontrolled-among others that persons not exposed to the program may be propelled by other life events into situations causing similar changes, and that even congruence of participation in a mental health program and change does not prove that the latter came about as the result of the former.

It follows from what has been said that I cannot feel that this study really tests a hypothesis, or conclusively demonstrates effectiveness (or if negative results were to be obtained a lack of effectiveness) of the program in question. It may surprise you to hear that despite these and other doubts, I strongly feel that studies of this kind are making an important contribution not only on a descriptive fact-finding level, but theoretically as well. The Potential contribution of such evaluation programs seems to me to be the


following. By explicitly building our assumptions, observations, expecta- tions and biases into the research design we are prevented from also testing the “truth” of these assumptions, etc. What we really do is to say: assuming these are relevant items of information correctly assessed, assuming that certain changes are correctly classified as improvements, and assuming that statistically significant differences associated with different programs represent the effects of these programs, what we actually observe can be meaning- fully interpreted within the framework of our explicit and implicit working hypotheses. Any deviation from results to be expected if our assumptions are correct directs our attention to what must be misconceptions on our part. In other words, in the mental health field there is need for a t least two kinds of demonstration. One would be the actual validation of the hypotheses on which clinical practices are built-which I think is not to be expected from this type of methodology. The other is to show systematically that whatever relevant information we are able to gather is compatible with our assumptions, that there is nothing in the phenomena and relationships among them to contradict our assumptions, or-to put it a third way-to establish the plausibility of our hypotheses rather than their probability. Proving our hypotheses well adapted to the phenomena they are meant to explain strikes me as a research task of fundamental importance, and as a step which must precede any rigorous validation study. This first and crucial test of the use- fulness of our theoretical orientation to the mental health problem, and of the clinical and educational practices that follow from it, is the important contribution to be made by studies of this kind.

Documents

METHODS FOR COMMUNITY MENTAL HEALTH RESEARCH