Upload
vincent-wilkinson
View
215
Download
2
Tags:
Embed Size (px)
Citation preview
Regional Educational Laboratory at EDCrelnei.org
1
Research Agenda Workshop
[District/State Name]
Session 1: [Date and time]
Session 2: [Date and time]
[Facilitator Name]
Regional Educational Laboratory at EDCrelnei.org
Welcome and Purpose
Our purpose in this workshop series is to • Engage in a collaborative process to identify district/state
priorities, and • Develop a set of research questions that comprise a
coherent research agenda that will be the work of our district/state for next 3-5 years.
2
Regional Educational Laboratory at EDCrelnei.org
Goals for the Session
• Session 1 (today) will focus on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We will leave today having prioritized initial research topics.
• Next session, we will generate research questions, and refine and prioritize our identified topics and research questions. We will leave with an initial research agenda for our district/state.
3
Regional Educational Laboratory at EDCrelnei.org
Agenda for the Session
• Introduce workshop participants• Investigate types of research and levels of evidence,
including reviewing the pre-work and additional research examples
• Discuss examples of researchable questions and research agendas
• Review district/state goal and generate some possible topics
• Prioritize possible research topics• Closing
4
Regional Educational Laboratory at EDCrelnei.org
Introductions
• Please share your:– Name– Agency/where you work– Your role there
– [if there are pictures of the participants you can insert an additional slide with the name and pictures while introductions are taking place].
5
Regional Educational Laboratory at EDCrelnei.org
6
Primary Types of Research
• Different types of research allow us to answer different types of questions and to make different types of claims.
• What’s Happening– Descriptive
• What’s Known– Descriptive
• Making Connections– Correlational
• Making an Impact– Impact/Causal
Regional Educational Laboratory at EDCrelnei.org
7
What’s Happening
• Descriptive studies do not examine cause and effect but look at what is happening: trends; baselines; and experiences of individuals, groups, or programs– What is the trend in science NECAP scores over past 5 years?– What are the certification characteristics of beginning teachers in
the region?– What are the features of district teacher evaluation systems?
• Methods tend to include secondary data analysis, document and records review
• Tend to use descriptive statistics: averages, frequencies, and percentages
Regional Educational Laboratory at EDCrelnei.org
8
What’s Known
• Another type of descriptive study focusing on reviews of the literature and other research that has already been conducted.– Literature review– Meta-analysis
Regional Educational Laboratory at EDCrelnei.org
9
Making Connections
• Correlational studies look at the relationship between two or more variables but one cannot infer causality. – How do math course taking patterns vary race and gender?– What factors are related to differences in graduation rates across
districts?
• Methods tend to be secondary data analysis of administrative or other state, district, or school datasets
• Statistical methods include: differences in means, ANOVA, multivariate regression models
Regional Educational Laboratory at EDCrelnei.org
10
Making an Impact
• Causal/Impact studies usually examine questions about impact, building on descriptive studies– Does this afterschool program increase student engagement?– Does this math intervention lead to improved math achievement?– Do new teacher evaluation policies improve student achievement
outcomes?
• Many “opportunistic” possibilities in implementation• According to IES, an impact study:
– examines the effectiveness of a particular policy, program, or practice– can be small-scale or large-scale– can be used in a formative way (e.g. a short-term pilot study) or a
summative way
Regional Educational Laboratory at EDCrelnei.org
11
Making an Impact
• Only certain types of designs can be used to assert causal claims or claims about program impact
• IES-funded impact studies must meet What Works Clearinghouse standards– Randomized Control Trials (RCTs) = “the gold standard”– Randomization of treatment and control groups (e.g. those who
get the intervention and those who don’t)– Other designs, including quasi-experimental, matched
comparisons or others may not yield unbiased estimates of impact
Regional Educational Laboratory at EDCrelnei.org
12
Investigating Research Examples
• Refer to the 2 research summaries you read during pre-work
• You will be placed into breakout rooms where small groups will focus on one study and discuss – Research questions– Data sources– What was answered/found– Study design—discuss and add this information– What else there is to learn/ future research questions
Regional Educational Laboratory at EDCrelnei.org
13
Discussion: Research Design
• Back in the large group we will share out from our small group discussions
• What were the studies about? • What is and is not answered by research
questions/designs?– What related research questions might you ask?
Regional Educational Laboratory at EDCrelnei.org
14
Researchable Questions
• What is a researchable question and is not?– Reasonable– Appropriate– Answerable– Measurable
• How do you develop researchable questions?– Questions, concerns, and values of stakeholders/community– Important issues raised in the field or literature– Professional standards or guidelines– Views and knowledge of experts– One’s own views and judgement
Regional Educational Laboratory at EDCrelnei.org
15
Research Agendas
• What is a research agenda? • What does a research agenda look like?
– Linear– Topical– Examples
Regional Educational Laboratory at EDCrelnei.org
16
Example: Research agenda that is less coherent
• TOPIC: COLLEGE READINESS– Descriptive study of high school dropout rates for key
subgroups– Descriptive study of policies for assigning students to
college math vs. general math– RCT of the impact of classroom coaching for new high
school English teachers in 5 large urban districts
(Source: REL Kickoff Meeting)
Regional Educational Laboratory at EDCrelnei.org
17
Example: Coherent, linear research agenda
• TOPIC: HIGH SCHOOL GRADUATION• Action Goal: Increasing the percentage of District X
students who graduate from high school– Descriptive study of high school dropout rates for key
subgroups– Descriptive study of re-enrollment rates in traditional high
schools and education trajectories of re-enrollees– RCT of the impact of alternative programs for degree
completion on high school graduation(Source: REL Kickoff Meeting)
Regional Educational Laboratory at EDCrelnei.org
18
Example: Coherent, topical research agenda
• TOPIC: MATHEMATICS LEARNING• Action Goal: Increasing mathematics achievement in the
Northeast region– RCT of the impact of student access to Algebra I
in grade 8– Descriptive study of math education practices for students with
disabilities in six schools in two Northeast states– Descriptive studies of math performance patterns for students
with disabilities in MA and NY
Regional Educational Laboratory at EDCrelnei.org
Our Process to Identify Topics
• Work from district/state goal to identify topics and narrow focus to two topics
• Identify subtopics and questions within top two topics• Prioritize questions [or subtopics]
19
Regional Educational Laboratory at EDCrelnei.org
Example: Gender in Schools Research district/stateGoal: Provide research that informs and promotes gender equity in schools
20
Topic Subtopic Questions TimingGirls in STEM Tracking achievement
differences
Encouraging interest and enrollment in STEM
Persistence in STEM majors
Attainment for BoysStructured Inequality
Regional Educational Laboratory at EDCrelnei.org
Moving from Subtopics to Questions
21
Subtopic Questions Timing
Tracking achievement differences
What are gender differences in STEM achievement in K-12 and how have they changed over time?Do gender differences in STEM achievement vary among districts and schools?
Encouraging interest and enrollment in STEM
What can teachers do?Are single-sex schools and classrooms better?What programs help promote STEM for girls?
Persistence in STEM majors
Regional Educational Laboratory at EDCrelnei.org
Moving from Subtopics to Questions
22
Subtopic Questions Timing
Tracking achievement differences
What are gender differences in STEM achievement in K-12 and how have they changed over time?Do gender differences in STEM achievement vary among districts and schools?
Encouraging interest and enrollment in STEM
What can teachers do?Are single-sex schools and classrooms better?What programs help promote STEM for girls?
Persistence in STEM majors
A fully specified research question
A general question that can be used to develop a set of research questions
Regional Educational Laboratory at EDCrelnei.org
23
District/State Goals
Regional Educational Laboratory at EDCrelnei.org
24
Goals & Research Topics
• Reflecting on our goals, brainstorm topics that are important to you and that fit under the goals.
• Record in your workbook on page 5.– If you are unsure whether a topic fits under the goal,
include it in the “possible” column of the page.
• The ideas don’t need to be refined, but we will be narrowing the list later in the session, so focus on topics that are most important to you.
• You will have 5 minutes to think and write by yourself.
Regional Educational Laboratory at EDCrelnei.org
25
Research Topics-Breakout Rooms
• You will be placed into a virtual breakout room to discuss the topics you brainstormed (15 minutes)
• Share your topics with your small group• On your breakout room whiteboard:
– Write the topics that you would like to bring back to the whole group for consideration as priority topics
• These topics will be brought back to the whole group, when the breakout time ends.
Regional Educational Laboratory at EDCrelnei.org
Grouping our topics
• Together, we will engage in a process of grouping the topics.
• Our identified topics are diverse but some are related.• What topics generated in the groups do you think go
together?• After we review the grouping of topics we will name the
topics.– The name becomes the “topic” and the items grouped under that
topic name are “subtopics”
26
Regional Educational Laboratory at EDCrelnei.org
27
Narrowing the List of Topics
• The Goal is to narrow the list of topics to 2 priority topics for our work—we will do this using a modified Focusing Four process– Brainstorm: We did earlier and grouped our ideas into topics– Clarify: We will review the groupings.
• We can clarify what any topic is, suggest moving topics or splitting groupings.
– Advocate: Think about narrowing to 2 topics that could comprise a good agenda when taken together.
– Canvass: You will each have 3 votes to use to indicate the topics that you think should be a research priority for our district/state.
Regional Educational Laboratory at EDCrelnei.org
28
Clarify
• Review the “groupings” of topics• Do any topics on any list need to be clarified?• Do the groupings make sense? Should we move any
items or split any groupings?• Can anything be combined or removed?
Regional Educational Laboratory at EDCrelnei.org
29
Advocate
• Think about which cluster of topics, when taken together, provide the best research direction for the district/state.
• Share your preferred priority topic and provide a few positive reasons why you think the topic grouping should be a priority for the district/state.
Regional Educational Laboratory at EDCrelnei.org
Break
5–10 minute break
30
Regional Educational Laboratory at EDCrelnei.org
31
Polls
• Vote once in each Poll to indicate your priority research topic for the district/state.– You can vote for the same topic in each poll if you like, or vote
for different topics.
• At the conclusion of the canvassing, we will review the results of our Polling and make some decisions about our 2 topics. – This will be your take-away from Workshop session 1.
Regional Educational Laboratory at EDCrelnei.org
32
Prepare for Next Session
• For one or both of the topics, think about:
– What research questions interest you under the priority topic areas? (these do not have to be at the level of researchable questions but they should be questions that are a finer grain of detail than the topics themselves). Email questions to facilitators before Session 2.
– Begin to think about the data sources that are appropriate to answering these questions and where they reside, if they exist.
Regional Educational Laboratory at EDCrelnei.org
33
Thank you
• Thank you for your participation and engagement today.• See you [INSERT DATE AND TIME OF NEXT
SESSION]!
Regional Educational Laboratory at EDCrelnei.org
34
Research Agenda Workshop Session 2
[District/State Name]
Session 1: [Date and time]
Session 2: [Date and time]
[Facilitator Name]
Regional Educational Laboratory at EDCrelnei.org
Welcome and Purpose
Our purpose in this workshop series is to • Engage in a collaborative process to identify district/state
priorities, and • Develop a set of research questions that comprise a
coherent research agenda that will be the work of our district/state for next 3-5 years.
35
Regional Educational Laboratory at EDCrelnei.org
Goals for the Session
• Session 1 focused on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We left with some prioritized topics.
• This session, we will:
– Generate, refine and prioritize research questions, and
– Leave with an initial coherent research agenda
36
Regional Educational Laboratory at EDCrelnei.org
Agenda for the Session
• Review topics from session 1• Generate questions (review questions sent in by email
and generate additional questions)• Prioritize questions
– We will prioritize all the questions generated in our two topic areas.
• Generate preliminary research agenda• Next steps
37
Regional Educational Laboratory at EDCrelnei.org
38
Review Research Topics and Begin to Generate Questions
• Highlights of topics from last session and review of research questions sent in by email.
• In virtual breakout rooms, work on the white boards for 30–40 minutes to generate questions related to the priority topic (and any subtopics within it)
• Groups should think comprehensively within their topic/subtopic
Regional Educational Laboratory at EDCrelnei.org
39
Share Out Small Group Work/Refine Questions
• Re-group into whole group and share white boards from small group work.
• Discuss questions generated and refine by:– Edit, add, and/or delete questions– Check: any additional questions—is something
missing?
Regional Educational Laboratory at EDCrelnei.org
40
Prioritize Research Questions
• Focusing Four process– Brainstorm: Review our list of questions, generated earlier– Clarify: Do any questions on the list need to be clarified? – Advocate: Think about narrowing a few questions in each
topic/subtopic area that could comprise a good agenda
– Last step will be done after the break
Regional Educational Laboratory at EDCrelnei.org
Break
5–10 minute break
41
Regional Educational Laboratory at EDCrelnei.org
Poll
• Poll: Vote for the research questions. – Vote once in each Poll– You can vote for the same question in each Poll or
different questions
42
Regional Educational Laboratory at EDCrelnei.org
43
Preliminary Research Agenda
• Review identified priority topic, subtopics, and questions• How many top rated questions should be on the
agenda? Consider:– Types of work (data alignment, workshops, studies, etc.)– Sequence– Short- and long-term nature of work
• Remember…
Regional Educational Laboratory at EDCrelnei.org
44
Research Agenda- continued
• Refinement of the research agenda should continue over time.
• The agenda should: – encompass the work that is most important to the district/state, – Align with the district/state goal and– Be the work that the district/state will do over the next 3-5 years.
Regional Educational Laboratory at EDCrelnei.org
45
Next Steps
• Continued refinement of agenda• Involvement in prioritization and study development• Work to conduct research or identify research support• Use agenda to recruit research and funding partners for
this work.
Regional Educational Laboratory at EDCrelnei.org
46
Thank you
Thank you for your participation and engagement today.