Loyola Marymount UniversityProgram Evaluation

Project REAL

Retaining highly qualified teachers who provide Evidence-based instruction,

Access to general education curriculum, &

Leadership in special education advocacy

LMU Special Education Program

• Small program with 2 full time faculty & 1 clinical faculty– Dr. Victoria Graf, Program Director– Marianne Mitchell, Program Assistant Director – Dr. Tisa Aceves, Project REAL PI– Hasmig Balian, Project Assistant

• Candidates take courses outside of our program & department

Project REAL Objectives

Objective 1: To restructure & redesign existing course content & field experiences to integrate competencies reflecting EB instructional & assessment practices

Objective 2: To help ensure candidates acquire the necessary skills to provide leadership in special education advocacy

Project REAL Objectives

Objective 3: To prepare candidates to provide appropriate access to the general education curriculum for all learners

Objective 4: To provide professional development for general & special education faculty

Objective 5: To develop & maintain an effective, rigorous mentoring & induction program that increases candidates’ successful completion of the program while developing their professional expertise & commitment to the profession

Major Changes to Program

1. Complete course review & development to reflect EB practices & new CA standards

2. Implement developmental sequence of courses3. Coordinate courses across departments4. Develop website to support candidates5. Shift away from evaluation of candidates to

mentoring of candidates6. Provide peer mentors 7. Infuse content reflecting Universal Design for Learning8. Infuse content for Special Education Advocacy

Program Evaluation Measures

• Special Education Program Survey• Observation Template• Revised Course Syllabi Evaluation• Signature Assignments (Livetext)• Workshop evaluations• Needs assessment (faculty, instructors, field

supervisors)• Course Evaluations• University Supervisor Evaluation by Candidates

Special Education Program Survey

• Drafted & administered survey November / December 2008 via Survey Monkey

• Surveyed completed by: – Alumni– Candidates– Supervisors– Instructors – Employers

(N=200)

SPED Program Survey - Sample Items1 = Sufficient attention is provided2 = Some attention needed3 = Moderately more attention needed4 = Significantly more attention needed

_____ 1. Developing the IEP and participating in the IEP process_____ 2. Classroom organization and management (routines, schedules, etc.)_____ 3. Whole-group behavior management (rules, consequences, etc.)_____ 4. Providing appropriate interventions for severe behavior problems_____ 5. Lesson planning and delivery_____ 6. Differentiation and individualization of instruction_____ 7. Evidence-based Instructional methodology_____ 8. Addressing the needs of English Learners with disabilities_____ 9. Providing culturally and linguistically appropriate instruction _____ 10. Conducting and interpreting assessment

Part II Qualitative Questions

SPED Program Survey

• Identified areas of strength & weakness

• Helped confirm annual exit interview comments from candidates

• Contributed to course changes

RelativeStrengths

Relative Weaknesses

Behaviormanagement

Lesson planning

Conducting & interpreting assessments

Differentiation

Understanding special education law

Interventions for severe behavior problems

Developing collaborativerelationships

Adapting curriculum

Observation Template• Observation template reflects the evaluation

of candidates in the field in the areas of Instruction, IEPs/Collaboration, Assessment, Professional Behavior (formative & summative tool)

• Contributors included: program faculty, faculty from Dept. of Clinical Ed., external evaluator, university supervisors

• Measure will be piloted with candidates this spring

Observation Template (sample items)C. Instruction 0 1 2 3 4

Ensures that students are engaged in learning activities. Comment:

0 = Not observed / insufficient evidence

1 = Does not meet expectations / practice not consistent with standard

2 = Is beginning to meet expectations/developing practice

3 = Meets expectations /maturing practice

4 = Is highly effective in meeting expectations / practice that exemplifies standard

Communicates the learning objective, purpose of the lesson, and learning expectations. Comment:

Communicates directions & instructional procedures of lesson in multiple ways.Comment:

Provides multiple opportunities & time for students to practice & apply what they’ve learned.Comment:

Draws on students’ background, interests & dev. learning needs for instructional planningComment:

Uses EB instructional techniques that reflect an awareness of different student learning styles. Comment:

Differentiates lesson using EB instructional strategies that minimizes barriers & expands learning opportunities for all learners (e.g., assistive technology).

Comment:

Demonstrates knowledge & understanding of the appropriate use of technology for information collection, analysis, & management in the instructional setting.

Comment:

Creates environment that removes potential instructional barriers thereby maximizing access to content.

Comment:

B. Classroom Environment/Behavior 1. Provides, creates and maintains positive, supportive classroom climate that

is safe and conducive to learning.Comment:2. Establishes, and maintains consistent standards of classroom behavior.

Addresses all behavior issues in a consistent manner (classroom level).Comment:

3. Manages individual student behavior effectively and in a positive, constructive manner (individual student level).Comment:

4. Establishes efficient classroom routines. Examples include transitions, beginning lessons, and general classroom routines.Comment:

Observation Template (sample items)

Revised Course Syllabi Evaluation• Program alignment of 325T project/program

objectives with new CA standards & LMU Conceptual Framework

• Program evaluation of previous program against new CA standards (0-2 point system)

• Determine EB practices through review of literature

• Advisory Board evaluation of new program syllabi– Evaluation Part I (special education courses) to be

completed mid January

– Evaluation Part II (courses outside of department) to be completed beginning of February

Revised Course Syllabi Evaluation

• Training for Advisory Board Course Syllabi Evaluators– Powerpoint Overview with voice over

– Sample Syllabi & completed sample rubric with voice over explanations

– CD with readings & resources

– Copy of Cover Letter, 325T Abstract, new CA Standards, & new program course sequence

Course Evaluation Rubric

CTC Standards, Project REAL Objectives (REAL) & SOE Conceptual Framework (CF), Program Objectives (PO)

Mark an X under the following ratings (0-2) indicating how much evidence the course demonstrates a particular standard.

AWARENESS / KNOWLEDGE (Passive Reception of Information)

SKILL DEVELOPMENT/USE (Analysis / Synthesis)

APPLICATION IN FIELD

0 0.5 1 1.5 2Final Score

No evidence that the standard is included in the class syllabus

Standard is stated in class syllabus (e.g., course objectives, course description)

Standard is stated in syllabus and includes 1 or more of the following: • tests or quizzes of knowledge/awareness• Professor demonstration • Discussion •Lecture/PPT •Reading •Observation (Passive) •Media Use

Standard is stated in syllabus. Course includes readings, tests (skill demonstration), & assignments. Course ALSO includes 1 or more of the following possible projects demonstrating application of standard:• Lesson Plans• Student Activity during class • Journal Response • Case Study (analysis of data) • Simulations

Standard is stated in the syllabus. Course includes required readings, tests (skill demonstration)-projects-assignments. Course ALSO includes application of standard in the field (classroom or school):• Teaching a lesson• Conducting assessment • Participation in IEP • Case Study (assessment or instruction) • Conducting FBA

Write the final point value for each standard rated. (0-2)

S2:Pr/Eth Practice (REAL 2, CF 3)

Process for Completing Rubric

S2: Pr/Eth Practice (REAL 2, CF 3)

S3: Div.Learners (CF 1)

S4: Collaboration (REAL 2, CF 3)

S5: Assessment (REAL 1, )

S6:Technology (REAL 3)

S7:Transition

Process for Completing RubricAWARENESS / KNOWLEDGE

(Passive Reception of Information)0 0.5 1

No evidence that the standard is included in the class syllabus

Standard is stated in class syllabus (e.g., course objectives, course description)

Standard is stated in syllabus and includes 1 or more of the following: • tests or quizzes of knowledge/awareness• Professor demonstration • Discussion •Lecture/PPT •Reading •Observation (Passive) •Media Use

Process for Completing Rubric

SKILL DEVELOPMENT/USE (Analysis / Synthesis) APPLICATION IN FIELD

1.5 2Standard is stated in syllabus. Course

includes readings, tests (skill demonstration), & assignments. Course ALSO includes 1 or more of the following possible projects demonstrating application of standard:• Lesson Plans• Student Activity during class • Journal Response • Case Study (analysis of data) • Simulations

Standard is stated in the syllabus. Course includes required readings, tests (skill demonstration)-projects-assignments. Course ALSO includes application of standard in the field (classroom or school):• Teaching a lesson• Conducting assessment • Participation in IEP • Case Study (assessment or instruction) • Conducting FBA

X

Process for Completing Rubric

Final ScoreWrite the final point value for each standard rated. (0-2)

Rating: 2

Next Steps in Course/Program Evaluation

• Conduct Advisory Board evaluation

• Conduct External Evaluators evaluation– Leonard Baca, Ph.D. (Univ. of Colorado, Boulder)

– Norma Lopez Reyna, Ph.D. (University of Illinois, Chicago)

• Continue collaboration with other departments

• Submit syllabi for 325T Evaluation in April

Question for 325Ts

• How can we better connect program & candidate evaluation to K-12 student achievement?– Current forms of evaluation

• Signature assignments include evaluations & interventions of diverse students in the field

• 2-3 courses use AIMSWEB to assess & monitor student progress