Matching Efforts to Outcomes: Are you really sure?

Keynote Address for Nonpublic Special Education Programs Annual Conference

Hank Bohanon, Ph.D.

hbohano@luc.edu

http://www.luc.edu/cseit

Center for School Evaluation, Intervention

and Training (CSEIT), Loyola University, Chicago

Thank you’s

• Sarah Sebert and Paul Nijensohn (ISBE)

• Barbara Simms (ISBE), Illinois State Technical Assistance Center

• Kelly Raucher (ISBE), SEL

• Kathy Cox (ISBE), Illinois ASPIRE

• Dean David Prasse, Loyola University of Chicago

Additional Thank yous

• Dr. Pamela Fenning, CSEIT

• Sara Golomb, CSEIT

• Agnieszka Kielian, CSEIT

• Lisa Lewis, CSEIT

• Dr. Diane Morrison, CSEIT

• Audrey Shulruff, CSEIT

Goal and Objective

Increase participants awareness of how

program data decision making and

evaluation efforts can be integrated.

• Data process

• Data integration

• Examples

Data Process

Process

• What are your questions?

• What are your data sources?

• What reports do you need?

• Who needs access?

• What resources do you have?

Questions

• Align with targets/strategic plan• Align with state initiatives• Training objectives

• Evaluation questions:

1. If you train, do people implement?

2. Do people implement with fidelity

3. Do the interventions sustain?

4. What is the impact on your constituents?

Data

• Process– Instruments (e.g., fidelity tools)– What is your process?

• Outcomes– Performance data

• Curriculum-based measures• Classroom checklists• Office discipline referrals• Transition information (e.g., post-secondary)

Instrument development

• Item development (align with questions)– Systems, Practice, Data, Outcomes– Schoolwide, Classroom, Non-Classroom,

Individual

• Judgmental validity (Expert Judgment)

• Concurrent validity (With Reliable Tool)

Instrument development

• Pilot instrument

• Item analysis (factor analysis)

• Review and update

• Balance input with process

Access

• What types of decisions need to be made based on the data and who makes them

– Statewide personnel

– Administrators

– Direct service providers

Reports

• In what format do you need the data?– Graphs

• Specific instruments• Combinations (process and outcomes)

– Summaries & reports– Output files (flat/rectangular format)

• Common identifiers• Clean your data (SOP)

Resources

• The more you can draw upon existing resources, the less time it will take

Data Integration

Technology Data decision making

• Data inputs– Data already exists

• SOP for cleaning data– Additional data from direct assessments

• SOP for reliability

• Data integration– Requirements, common identifier– National Educational Technology Standards

(NETS)– Universal data system for educational data

Tips

• Data cleaning

• Reliability of data entry

Reports

• Static – Graphs, tables, etc.– based on questions

• Dynamic– Analysis based

• requires export of flat files

Decision Making

• Statewide– Summative and formative: Annual and Quarterly

reports

• Administration– Formative: Professional development, tracking data

collection, reports

• Direct Service– Diagnostic: How much of the interventions is in place,

what is the impact, and

what changes do you need to make?

Integration of Data

• Example of questions

• Sample instrument

• Sample report

ConsiderationsKey Components from Three-Tiered Intervention Programs

STUDENTS WHO RESPOND TO A CONTINUM OF SUPPORTS

Students who respond to intensive academic and behavior support

Students who respond to less intensive academic and behavior support

Students who would respond to effective core academic and behavior curriculum

National Standard

Schoolwide support

Group Support

1-7%

5-15%

80-90%

Individual Support

OSEP-PBS

Key Elements

• Systems– Administrative

Commitment– Priority for Staff– Representative Team – Audit of practices – Action Plan– Data System– Internal Coaching– External Coaching

• Practices– Based on evidence

• Data– Process and impact

• What and with whom?

Schoolwide Supports

• Identify expectations of the setting

• Evaluate implementation and evaluation of core curriculum

• Develop team/plan/support

• Directly teach expectations

• Consistent consequences

• Acknowledgement• Collect data

– Process, academics and behavior

• Communicate with staff

• On-going evaluation

Questions to ask

• What does your current system provide information that is:

– Efficient, reliable, dependable, user friendly

• Who needs to see your data? – Who are your stakeholders

• What is your timeline?

Resource

• Visit our center website for more information– http://www.luc.edu/cseit/learning.shtml

• Useful example of reports– http://www.pbisillinois.org/

• Example of coaching and data collection– http://flpbs.fmhi.usf.edu/coachescorner.asp

Thanks!