Design Development and School Improvement:Bridging Research and Practice with

Equity-Relevant Interventions in Local Contexts (working title)

A Practical Guide

Rick Mintropwith Mahua Baral, Elizabeth Zumpe, and John Hall

Leaders committed to educational equity and justice need to be guided by creativity, moral purpose, and idealism; yet, evidence plays a large role. Equity-relevant improvements are about what we value, but also about what works. We need philosophy and deliberate reasoning to clarify what we value.

Fortunately, as professionals, we can also draw from a developed knowledge base on effective schools and teaching that reveal factors associated with better student outcomes. This body of knowledge, however, is relatively unhelpful in telling educators what to do about specific, but enduring problems of practice. Equity-relevant design development marries the values and ideals of visionary leaders with the evidence that their efforts worked. Against the idea of implementation fidelity, it assumes the autonomy of participants. Less concerned about universal effectiveness or broad generalizations, it holds that context sensitivity is the base of all successful improvement, but that design principles can travel.

This book is meant to be a practical guide for educational leaders who strive to contribute to this knowledge.

The theoretical foundations of design-based leadership in educationMany of us associate the term ‘design’ with the shape of objects (for example, designer clothes, designer furniture, industrial design, or the design of your smart phone). We often find these objects pleasing. Harbingers of design-based thinking like Tim Brown (2009) and Tom and David Kelley (2013) argue that, in our work and our organizations, we all are, or can be, designers. Designers do not only design objects; they can also design forms of human interaction that may elicit similar satisfaction among users and observers.

Following this concept of design, in the year 2006, the Graduate School of Education at the University of California, Berkeley, started a new doctoral program for working professionals, called the Leadership for Educational Equity Program (LEEP), which

Prospectus of an Upcoming Book Published by Harvard Educational Press 1

experimented with design development methodology as the signature pedagogy for the practice-oriented educational doctorate. In creating this signature pedagogy, we consulted an extensive literature on design experiments, design studies, and design development.

The logic of design developmentA design connects an actionable problem to an intervention. The task is two-fold. On one hand, design developers support schools or districts by responding to urgent problems of practice that pose a design challenge in a specific context. On the other hand, they work in a systematic fashion towards designs that can be transferred from one organizational unit to another. Practicality, impact, and transferability are the three most important quality criteria.

Figure 1 below summarizes the design development logic. First, the problem of practice (PoP) is identified and a design challenge formulated. To address this PoP, design development requires a sound theory of action, a sort of model or conceptualization that predicts how to move from a problematic state to a desirable one. A theory of action is then used to develop an intervention, which is then implemented and evaluated to see if the predictions contained in the theory of action were correct.

Figure 1. Design development logic model

This logic model only displays one iteration of conceptualizing, designing, implementing, and evaluating an intervention – the likely extent of what is feasible for an education leader within a single academic year. The call for continuous improvement in education organizations is akin to understanding that a corroborated design would likely take several iterations, over a longer period of time. Across the iterations, the design can generate a set of practices that seem to work for an identified problem of practice, practices that give insight to leaders in other organizations wanting to address a similar problem of practice.

Decisions and designWhile many types of decisions are small and made in a flash of recognizing past experiences in present situations, some are bigger. Big decisions require leaders to ask, "What is the behavior I am trying to change; what causes the behavior; what learning or

Prospectus of an Upcoming Book Published by Harvard Educational Press 2

unlearning needs to occur for the behavior to change; what kinds of activities, tools, or structures elicit that kind of learning; how do I know that it has worked?” Decision-making for these bigger decisions rests on the problem-solving skills that are evoked in a design process.

The daily flow of work rarely allows practitioners to address the complexity that design-based problem solving requires. Leaders usually only have time and energy for the kind of problem-solving that is done quickly, using shortcuts: “Strategy X has always worked for me when I have encountered situation Y, let’s do it.” Or when under pressure, “People who I trust tell me that Strategy X or Program Y have promise or have shown to work, let’s try it.” These kinds of shorthand decision rules, called heuristics, are necessary when making decisions quickly, but they are not good guides for important strategic decisions about new directions, new programs, new organizational structures, or new professional development initiatives, because they can often be misleading.

For the most part, educational leadership in disadvantaged contexts is severely constrained by decisions made elsewhere. Thus, leadership at the local level often ends up being about making the best out of rather adverse circumstances. But this does not mean that we cannot make advances towards more equitable services or learning opportunities, in the action space we control or in the sphere we influence.

Identifying an actionable problem of practiceJust as we have many questions, only some of which are researchable, we are faced with many problems, only some of which are actionable. In finding a problem of practice (PoP), the question is not: what am I interested in or what captures my fancy, but it is: what are the most urgent development problems, encountered in the organization where I work, for which we need new solutions or improved approaches. Therefore suitable problems of practice are burning issues in your organization for which people in leadership positions want answers. Problems of practice specifically deal with practices that are entrenched in leaders’ own organizations. With a focus on leadership, problems of practice deal with the way adults deliver services.

In addressing a problem of practice, we should capitalize on all the forces moving towards humanizing education that we can, but we must also be realistic. Some big problems may be worthy causes of a professional career, but are not actionable within the span of time-constrained improvement efforts. This means we need to focus on a problem within certain feasibility constraints. For example, if the big problem is how to make work teams in schools commit to closing achievement gaps, then the actionable PoP may be how to move a ninth-grade mathematics team to develop a program that articulates grade-level instruction with remediation. Or, if the big problem is how to make instructional supervision and leadership more effective, the actionable PoP may be how to help principals give good feedback after informally observing an instructional sequence of fifteen minutes.

Prospectus of an Upcoming Book Published by Harvard Educational Press 3

Upfront, most changes take additional investment of time, energy, and intellectual and material resources. Design-based thinking guards against overload and churn and asks of the professional to think hard about the connection between our intentions and the human and material resources we have at our disposal to meet our goals. When we do this, and people around us notice that not only do we have strong values and ambitious goals, but also a keen understanding of how to get change done given people’s resources, knowledge, and energy, we can become transformative.

Developing a theory of actionAll design begins with intuition. All of us have some pretty good intuitions about our work; otherwise we would not have had the success that has come our way. We have experienced repetitive cause-effect relationships in our work. We have experienced what works and what does not. When we see a behavior that poses a problem, we just know by association what might work as a remedy.

A theory of action rests upon a diagnosis of a problem, a change process that is predicted to address the problem, and a clear picture of the desired outcome. We have intuitive ideas for each of these elements, but in order to make our improvement process more exact, we must move beyond the intuitive space into a sharper and more conscious analysis. Challenging our intuitive theory of action (and intuitive problem diagnosis, change process, and desired outcome) requires the help of knowledge that has been accumulated through educational research and professional practice.

Local needs assessmentBefore we jump into spending time on exploring the knowledge base for our PoP and thinking about designing interventions, we need to check if our assumptions are correct. Local needs assessments, often exploratory in nature, aim to question assumptions regarding the problem of practice; and, local needs assessments can also be used, along with consulting the knowledge base, to think about remedies or solutions. Our PoP should focus on beliefs, attitudes, or practices of adults that are members of, or associated with, our organization. Many of the beliefs, attitudes, and practices (in short, behaviors) that we think to be problematic are perceptible. We can observe them or we can hear about them through conversations. So in a first step of the problem diagnosis, we need to look for these perceptible behaviors - we look at symptoms. In this first round of problem diagnosis, we stay with low-inference descriptions of these behaviors (i.e., beliefs, attitudes, practices). We are not making high inferences yet, and we refrain from associating causes as of yet.

But we will run into limits of perceptibility that social scientists are all too familiar with. Practices are observable, but we often do not know what they mean without knowing the actors’ intentions, which are sometimes unacknowledged, subconscious, or concealed. For example attitudes (e.g., being inspired, motivated, committed, sympathetic, fearful, hostile, not liking certain people, pretending to agree, etc.) can only be inferred through speech or actions, and given the uncertainties with speech and action, attitudes can often only be

Prospectus of an Upcoming Book Published by Harvard Educational Press 4

ascertained through relatively higher inferences. In everyday life, we help ourselves with these high inferences by trusting our intuitions and by reading subconscious cues, such as facial expressions. But in design thinking, we will need something more robustly verifiable for our problem diagnosis. So, for all those high-inference states of mind we need clues of more low-inference behavior that people can display without directly disclosing what is going on in their minds. This is called operationalizing – making concrete these abstract high-inference constructs (e.g., motivation, creativity, engagement).

Consulting the knowledge baseProblem diagnosis moves from low-inference evidence of observable behaviors to high-inference descriptions of patterns of behavior and the causes of behavior. This shift involves investigating the underlying or root causes of the problem, or the factors that explain what contributes to the recurrence of a behavioral pattern or what holds it in place. Understanding these factors requires consulting the knowledge base to develop some theoretical knowledge of the macro-, meso- and micro- level dynamics that contribute to human behavior within schools.

Having consulted the literature to get a better sense of the causal factors underlying the problem, there is now a need to find out whether these causal factors are evident in the organization at hand before you begin to think about what kind of change process or intervention is needed. This is because this second round of local needs assessment data may shift one’s the understanding of the problem, which will mean that the search for solutions will therefore need to move in a different direction.

Diagnosing a problem is one part of a theory of action, but a diagnosis alone does not mean that we know how to treat or cure the disease. From the knowledge gained about the problem, we can move beyond our intuition in order to construct a theory of change that predicts how the problem can be remedied.

Theory of changeA theory of change spells out how we believe people will change beliefs, attitudes, or practices in the desired direction. It specifies the learnings that need to take place, the main drivers that we believe will motivate the intended change dynamic, and the problem-specific activities in broad strokes. It also addresses the issue of feasibility, since activities are carried out in specific contexts that afford assets and pose obstacles for implementation. As with the problem diagnosis, we need to gather ideas from professional sources, research, and professional experiences that help us understand what to do to get from Point A to Point B.

Changing the behavior of those who provide services in educational organizations is highly complex. It involves organizational structures and resources that distribute learning opportunities. It involves knowledge, skill, and cognitive understanding. It involves attention, priorities, concentration of work energy, motivation, goals, and demands. But most of all, it involves shared meanings, interpretations, expectations, norms, values, rituals, and routines that to a large part are tacit and subconscious and hold together an enterprise that is at its core full of ambiguity, inconsistency, and contradiction.

Prospectus of an Upcoming Book Published by Harvard Educational Press 5

At the core of change, people are unlearning unwanted beliefs, attitudes or practices, and learning new ones. Even when we change a rule, use a new instrument, create a new organizational structure, or allocate new resources, at the core, people need to learn what to do with these new conditions. From the problem diagnosis we already know what the unwanted behaviors are. But we do not know how to dislodge them, what to replace them with, and what learning processes need to unfold in people. Note that we are not yet talking about activities. We first need to know what learnings or unlearnings need to take place.

But knowing what needs to be unlearned and learned is only part of the equation. You also have to know how to get that process off the ground within an organization, and this is why you need to think about drivers. Drivers are the levers or catalysts that can make things happen within the organization. Some drivers are related to resources – reorganizing roles, tasks or procedures, allocating resources differently, using new technologies, and introducing incentives. Other drivers may be related to the power of leadership – goals, visioning or command and control. There may be professional drivers, as well – training, collegial reflection, and inquiry.

Once the problem is diagnosed, learnings and unlearnings are identified, and suitable drivers of change are selected, activities are planned accordingly. These activities work in specific contexts and rely on local knowledge and ideas. This means that the intervention must be designed in a way that takes the contextual conditions into account. This ensures that the design is feasible to carry out within a given time frame of the intervention. But it also means that, once a theory of change is broadly conceptualized, the design process needs to shift the focus back to the local context in order to check if the assumptions, now more thoroughly refined through consulting the literature, still hold.

Baseline After formulating the theory of action – the problem diagnosis, theory of change, and activities – but before planning the details of the intervention and getting underway for implementation, another round of needs assessment data is essential. In this case, the needs assessment data allows us to test our problem diagnosis at the beginning of the design experiment by providing a baseline assessment.

The baseline assessment is a much more succinct form of data collection than the previous rounds of a needs assessment, and is standardized so that the same observations, surveys, interviews, document analyses, or task analyses can be conducted at the beginning and at the end of the intervention. The baseline instruments should focus on low-inference behaviors (operationalizations) that indicate the current state of affairs on the main components of the design. As a needs assessment, the baseline assessment succinctly corroborates whether our previous thinking was correct. It also enables us to assess growth when we repeat the same collection of data at the end of the design.

Prospectus of an Upcoming Book Published by Harvard Educational Press 6

From theory of action to intervention designAn intervention design consists of a sequence of activities that together, or in combination, intervene in existing knowledge, beliefs, dispositions, or routines in order to elicit new learning leading to new practices. Activities could revolve around new inquiries, materials, tools, rules, procedures, or resources that foster opportunities for new practices to take hold. Interventions are designed. That is, interventions are formal activities planned for a defined period of time and for a specific organizational setting. Intervention designs are surface artifacts of underlying theories of action. They are not unlike a teacher’s lesson plan that lists surface instructional activities, but implies the teacher’s underlying idea of how students learn the new content.

Planning the intervention activities requires keeping distinct the proximal and distal goals. Bridging these two kinds of goals speaks to the value of incremental changes in addressing systemic social problems. While the systemic social problem may be the student achievement gap, this is a distal goal that cannot be met by a short-term design in a particular organization. But a designed intervention can take an incremental step towards the distal goal. It is this proximal goal of the intervention that needs to drive its design, implementation, and evaluation of impact. For example, in service of the distal goal of closing the achievement gap, a leader may design an intervention for the proximal goal of creating an organizational focus around the need for student academic engagement in the classroom. The impact of the intervention is not measured in its effect on the achievement gap, but its progress towards the proximal goal. But a sequence of interventions over a longer period of time should make a dent in the achievement gap.

Moving into the method spaceThere are many different ways to perceive reality. When we read a good novel, we know that the characters in that novel are fictitious, yet they teach us something about ourselves. They reveal a deeper reality to us that fascinates us, yet they are fiction. How can fiction reveal reality? A good writer uses techniques or methods of narration that make us identify with characters in her story. We identify because the characters and situations are recognizable to us, even when the scenery is making the familiar strange. The narration resonates with us at a deeper intuitive and emotional level. It is perceived as trustworthy, and its trustworthiness opens us up to the deeper insights that the narrator’s characters reveal: catharsis, the cleansing of emotions through higher discernments. Narrative techniques are the appropriate method for a novel.

This is not what we are after in design research. At the core, we as researchers are cool and detached. We really want to know what works, and it is the beauty of robust findings that makes us quiver. Narration and trustworthiness are part of the picture, but our “plot” revolves around a dispassionate connecting of logical and procedural dots. Our goals are not identification with heroes and catharsis, and we want to avoid fiction at all cost. Rather, we aim at visibility, verifiability, and transferability.

Prospectus of an Upcoming Book Published by Harvard Educational Press 7

Impact and process dataDesign development methodology consists of an ordered sequence of fixed steps that result in a researchable intervention design. An intervention design becomes researchable when it consists of a set of planned, prepared, and pre-arranged formal activities around which an accurate and efficient search for evidence of change and effect can be organized. The search for evidence proceeds with established or prescribed practices or methods of collecting, analyzing, and interpreting data that connect baseline, process, and outcome. Thus, one characteristic of design development methodology is that it requires us to be mindful of the precision of both the intervention design (a stream of change activities) and the research design (a stream of data collection activities) that may run parallel. Precision, however, does not mean inflexibility.

To read about the full design process, look out for the published book with chapters on: Implementing Designed Interventions Design Principles That Can Travel

ConclusionLeaders who engage in design-based thinking must themselves be open to learning, experiment, and critique. They need to base their interventions on testable assumptions based on evidence that can be disputed by participants. A design connects the dots between a problem of practice, an intervention that is constructed with a testable logic, and metrics that indicate to what degree the implemented design helped participants to express or fulfill identified needs. A design-based approach is appropriate in a work environment that increasingly demands of leaders to show how their decisions are based in research and evidence, and that they have produced results. These results need to be confirmable by participants and verifiable by a community of equity-relevant reformers.

Prospectus of an Upcoming Book Published by Harvard Educational Press 8