Report Writing Tips - cec.sped.org/media/Files/Standards/Professional Preparation... · knowledge and skill set(s) , and the performance levels within rubrics demonstrate the use

Reviewing and Report Writing

CEC Program Reviewer Webinar04 2011

Richard Mainzer

Kathlene Shank

Rachelle Bruno

4/14/2011 © 2011 Council for Exceptional Children. All rights reserved. 1

Agenda

© 2011 Council for Exceptional Children. All rights reserved.4/14/2011

Language to Reduce

Confusion

Review Report

Sections

Documentingthe Decision

Functioning as a Team

Questions and Future

topics

2

Language to Reduce Confusion

© 2011 Council for Exceptional Children. All rights reserved.

1. CEC Content Standards2. Major Elements3. Informed by Specialty Set(s)4. Preponderance of the evidence

4/14/2011 3

Clarifying LanguageThe major elements of each CEC Content Standards informed by the specialty set(s) are to be addressed in the assessments, scoring guide/rubrics, and data tables.

….aligned to the major elements of the CEC Content Standards

as informed by the specialty set(s)

© 2011 Council for Exceptional Children. All rights reserved.4/14/2011 4


CEC Preparation Standards: Structure

Major Elements

Content Standards

Specialty Sets

Literature Reports

• Align to• Assessment, • Scoring Guide and• Data Tables

Major Elements

• Rich narrative for all special educators• Informed by the appropriate specialty

set(s

Content Standards

• Common Core Knowledge and Skills • Plus specialty specific knowledge and

Skill• Used for Curriculum and Program

Development

Specialty Sets

• Literature base for each knowledge and skill

• Classified

Literature Reports

5

Specialty Sets Inform the Content Standards

CEC Content Standards as informed by the appropriate specialty knowledge and skill set are to be reflected in the program’s assessments, rubrics, and data. Every program must demonstrate alignment to the major elements of the CEC Content Standards whether it be the Initial or Advanced Content Standards. However, without the appropriate specialty knowledge and skills set, every special education preparation program would look the same. Since this is not the case, knowledge and skills specialty sets are used to differentiate programs.CEC Reviewers expect that the assessments, rubrics, and data submitted by a program are informed by the appropriate specialty area knowledge and skill set(s). Programs can do this in a variety of ways, but the most meaningful way is to assure that assessment items address appropriate content, issues, and populations within specialty knowledge and skill set(s) , and the performance levels within rubrics demonstrate the use of the appropriate content, issues, and populations within specialty knowledge and skill set(s). There is no expectation for explicit and complete correspondence between the items in a knowledge and skill set with assessment items, and reviewers do not look for this level of correspondence. Likewise, reviewers do not expect that programs use the exact wording of the knowledge and skills within the rubrics. However, it is expected that the appropriate specialty knowledge and skill set is used in designing assessments and rubrics.



Where does the program report document how the specialty set informs the major elements of the CEC Content Standards?Assessment Use the specific content, populations, and issues

from the specialty set are used throughout the assessment items and components.

Scoring Guide/Rubric

Use the specific content, populations, and issues from the specialty set are used throughout.

Section I Narrative

Explain how the assessment addresses the content, issues, and populations

Program reports not required to cite any specialty set items.

4/14/2011 7


Question What does CEC mean by “a preponderance of the evidence” for the major elements of the CEC Content Standards?

Response

“Preponderance of evidence” is a standard of proof indicating that the evidence is clear and convincing, as opposed to the more rigorous “beyond a reasonable doubt” standard. CEC program reviewers use the “preponderance of the evidence” standard with each of the CEC Content Standards. Using a preponderance of the evidence standard, the reviewer judges whether the evidence in the report is clear and convincing. A preponderance of evidence cannot be reduced to a simple quantity, i.e. 75%. It is a reasoned judgment by a set of collegial reviewers and auditors based on the evidence presented.In order to determine that a program meets a CEC Content Standard, the reviewers judge whether the pieces of evidence presented in the program report are clear and convincing that the program assessment aligns with the major elements of the respective CEC Content Standard and that the program data demonstrate that the program candidates are mastering the major elements of the CEC Content Standard.For a program report to receive a “Met” decision overall, the reviewer must find that every CEC Content Standard is “Met”. The reason for this last criteria is based on the NCATE Option C.

Preponderance of Evidence

8

Review Report Sections


Part A. Recognition Decision Part A is actually done near the end of your

review Make a decision If the decision is “Nationally Recognized with

Conditions” then be sure to include all the conditions

Be sure that all comments in the Program Review Report support the team decision

If the team can not make a decision, refer the program report to the CEC Audit Committee as a last resort


Part B. Status of Meeting CEC Content Standards

Rate each CEC Content Standard and the CEC Field Experience as Met (M), Met with Conditions (MC) or Not Met (NM)

For each CEC Content Standard and the CEC Field Experience include an explanation of the basis for the rating Explanations should provide enough information to

support the rating Name the assessment by name and number that

provides evidence for the given CEC Content Standard. Please address each assessment cited by the program.


Field ExperiencesAppropriate to the license or roles for which they are preparing, is the evidence clear and convincing that the candidates progress through a: series of developmentally sequenced field

experiences for the full range of ages, types and levels of abilities, and collaborative opportunities that are supervised by qualified professionals.


Findings and ConclusionsDifferentiate your “findings of fact” from your conclusions and write conclusions in identifying the items that must be addressed.


Finding of Fact The alignment of the rubric for assessment 5 with majorelements of CEC Content Standard x is not clear.

EvaluationConclusion

The assessment 5 rubric must be clearly elaborated to alignto the major elements of the CEC Content Standard asinformed by the specialty set.

4/14/2011 13

For each CEC Content Standard


Identify the program assessments by number and name that the program report aligns to the CEC Content Standard

The program report cites Assessment #1 (Praxis), Assessment #2 (Course Grades), and Assessment #7 (Intervention Assistance Team Project) as evidence in meeting the major elements of CEC Content Standard 1.

Describe your findings for the assessment, scoring guide, and data for each of the assessments identified , i.e. what do you consider as facts

Assessment #1 (Praxis) has elements aligned with the major elements of CEC Content Standard 1, but it is not sufficient evidence by itself

Evidence is not presented that the assessment and rubric for Assessment #2 (Course Grades) aligns with the major elements of CEC Content Standard 1.

The evidence for course grades does not include delineated rubrics that can be disaggregated to the major elements of the CEC Content Standard, the evidence based on course grades is lacking across the respective CEC Content Standards.

Evidence is not presented that the assessment and rubric for Assessment #7 (Intervention Assistance Team Project) aligns with the major elements of CEC Standard 1.

Without appropriate alignment, it is not possible to judge the whether the candidates are mastering the major elements of the CEC Content Standards.

State the conclusions The assessment and rubric for Assessment #2 (Course Grades) and for Assessment #7 (Intervention Assistance Team Project) must be delineated to align with the major elements of CEC Standard 1.

Based on the evidence, CEC Content Standard 1 is not met.

4/14/2011 14

Part C. Evaluation of Program Evidence

C.1: Candidates’ knowledge of CEC Content Standard

C.2 Candidates’ ability to understand and apply pedagogical and professional content knowledge, skills, and dispositions.

C.3: Candidate effects on student learning and creation of environments that promote student learning


Part D. Evaluation of Use of Assessment Results

Evaluation of Section V of Program Report Is the evidence clear and convincing that the

assessment results are used by the program faculty in evaluating the program, counseling candidates, and revising courses or other elements of the program?

Has the program faculty made or considered making program changes based on assessment evidence?

Are the faculty interpretations consistent with the evidence provided in the program report?

Are the implications for programs that appear in this section of the program report derived from the interpretations?


Part E: Areas for Consideration A program can only be Nationally Recognized

(without conditions), when ALL CEC Content Standards and the field experience standard have been met.

If program decision is “Nationally Recognized with Conditions” be sure “Areas for Consideration” are consistent with what is stated as the “Conditions” under each CEC Content Standard

Do not give opinions or advice, but rather state what needs to be corrected.


Part F: Additional Comments F.1: Comments on Context or other issues F.2: Instructions for the Board of ExaminersCould be issues not related to the CEC

Content Standards, but related to the NCATE Unit Standards


Documenting the Decision


Program Review DecisionsCEC Program Reviewers must make one of the following decisions based on the program report:

The program is nationally recognized.

The program is nationally recognized with conditions. Insufficient assessment descriptions, scoring guides, or data

Insufficient alignment of assessments, scoring guide/rubric, and data to the major elements of the CEC Content Standards as informed by the appropriate specialty set

Insufficient data

Insufficient results for the 80% rule for Standardized Test on Assessment #1

The program is not nationally recognized; a Resubmission Program Report must be submitted. Resubmission Program Reports are NOT full reports.

They only address the identified conditions, so be sure to state what needs to be submitted in the Resubmission Program Reports clearly.


Program Report Options A-D

CEC program reviewers will only review Option A Reports for now


A Initial Program ReportsB Assessment flexibilityC Data onlyD Research

Program Report Option C

Since “Nationally Recognized” programs choosing option C in the next review cycle submit ONLY performance assessment data. A “Nationally Recognized” decision

now means that the ALL CEC Content Standards and the field experience standard are “met”


NATIONALLY RECOGNIZED WITH CONDITIONS

One of the difficulties CEC program reviewers have had is not having sufficient the information in a “response to conditions” report. To help program report developers know specifically what to resubmit in resubmission program review reports”, use the following italicized language in Part A. Recognition Decisions of Program Review Reports.


NATIONALLY RECOGNIZED WITH CONDITIONSFor each CEC Content Standard or CEC Field Experience Standard found to be either “not met” or “met with conditions”, the program resubmission report must provide: The Section II and Section III tables that document the alignment of

each program assessment to the major elements of the CEC Content Standard as informed by the appropriate specialty set(s);

The assessment descriptions, scoring guide/rubric, and data for each of the program assessments that provide the evidence that they are aligned to the major elements of each of CEC Content Standard as informed by the specialty area knowledge and skills set(s); and

Sufficient performance data for reviewers to determine that the preponderance of the performance data for each of the CEC Content Standard as informed by the appropriate specialty set(s) demonstrate that the program candidates master the major elements of the CEC Content Standards as informed by the appropriate CEC knowledge and skill set(s).



Whenever a “recognized with conditions” program report has additional conditions, these specific conditions should be added to the review report.

The assessment description, scoring guide/rubric, and data form a vital chain and is only as strong as the weakest link. If one or two of the links are the basis of a conditions decision, it will be helpful to program faculty if the review report specifically identifies the weak link(s). For example, “While all the materials described above are required in the

resubmission, the alignment of the assessment and rubric for assessment 5 (nnn) was not clearly aligned to the CEC Content Standard. Assessment 5 (nnn) and its rubric must clearly align to the major elements of the CEC Content Standard as informed by the specialty set.” This lack of alignment to the major elements of the CEC Content Standards was consistent for Assessment 5 (nnn).



Well-written “Nationally Recognized with Conditions” decisions are:

CLEAR SPECIFIC OBJECTIVE CEC Content STANDARDS BASED EVIDENCE BASED Consistent with all narrative in the Review Report Complete, i.e. SPECIFICALLY IDENTIFIES

EVERYTHING that must be addressed in the next report



A program is eligible for “Nationally Recognition with Conditions”

with no data


New Data Requirements


Initial Program Reports

Two applications or cycles of the assessments

Resubmission Program Reports

One application of new or modified assessments

First time NCATE accreditation institutions

One year of data on the key assessments

State test Two applications of data equivalent to data collected in two semesters

28

Writing Style Use formal professional writing style Do not use “I” or “me” or any other first

person language Do not try to tell program faculty how to

solve a problem Do not make side comments or direct

questions to program faculty or CEC Audit Team


Mechanics are Important Write in complete active-voice sentences. Be careful cutting and pasting to be sure it

fits the particular program about which you are reviewing.

Check spelling.


Completing the Report Thoroughly

Write comments that support the rating for each CEC Content Standard

Name assessments by name and number, never by number only

Write narrative for each section and each part excluding only “Directions to the BOE” for which reviewers may or may not choose to respond


Decision Must Be Supported The report should support the decision Areas for considerations should tell program

faculty what needs to be done, not how to do it If the decision is “Nationally Recognized with

Conditions”, all conditions must: Be stated in a as a conclusion that tells program

faculty what must be corrected and submitted in next report; be sure it is everything the next set of reviewers will need

Identify for program faculty any patterns of specific “weak links”.


Further suggestions Do not offer prescriptive suggestions Avoid overstating strengths Avoid personal observations Use professional language If doing more than one report for an institution be

careful when you cut and paste to be sure you are using language specific to the program you are reviewing

Proof, proof, and then proof again!


Can’t make a decision?Refer to the Audit Committee as soon as possible and please communicate why you can not make a decision


Functioning as a Review Team


Initial Tasks Open institution submission in the PRS Check attachments (Are all assessments there?

Do they open?)

Insure that CEC standards have been addressed not another SPA.

Notify NCATE immediately if there is a problem with opening or a clear mistake.


Team RolesReviewer(s)

Communicate with the lead reviewer concerning questions and issues with the program report

Participate in the decision making process Write and submit a reviewer report according to the agreed upon timelines

Lead reviewer Contact reviewers for your team Set timelines Respond to team members’ questions Assure that all members have composed and posted individual reports (even if

they are not complete reports) Assure all members have contributed to the final report Facilitate decision making and arrive at consensus Synthesize reviewers’ comments into a final report Post the final team recognition report.

Audit Team Reviews every Program Recognition Report to assure decisions across reports

are consistent Reviews institutional reports when team can not reach a decision Review reports that have been flagged by NCATE staff


Be a Team: Use Consensus

Communicate The “Lead” should initiate and facilitate

communication but everyone on the team is responsible for communicating

Any submission problems contact NCATE promptly


Wearing BlindersThe revised team that looks at the program when it is re-submitted will only review the program based on the “conditions” and determine only whether these are met or not

High Stakes!


Timelines and Difficulties Please make every effort to meet the

report submission timelines If you have any difficulties submitting a

report or need other help with the PRS system, contact Elizabeth Vilky at NCATE(202) [email protected]


mailto:[email protected]�

Questions and Comments


Future Webinar Topics


This process is important and YOU ARE APPRECIATED….


Documents

Report Writing Tips - cec.sped.org/media/Files/Standards/Professional Preparation... · knowledge and skill set(s) , and the performance levels within rubrics demonstrate the use