NAGC Reviewer Refresher

Reviewing Teacher Preparation Programs in Gifted and Talented Education Using

the NAGC-CEC Teacher Preparation Standards

Reviewer Responsibilities

NAGC is grateful to reviewers for volunteering to:

• Commit personal time to review electronically one program report per cycle.• Write individual report findings and post on the CAEP’s AIMs

system.• Communicate and cooperate with teammates to schedule

time to discuss findings and write a final team Recognition Report.• Submit quality reports and meet CAEP-established report

deadlines.• Maintain confidentiality of the review and any comments

about the program made in discussions or in final report.

Steps for Completing a Program Review1. Open Program Report and Read Sections I – V. 2. Analyze the quality of the report by examining the alignment of

the report and its attachments to the standards.

THIS IS THE HEART OF YOUR WORK but note that it can be slow going to review each assessment, scoring guide, and data chart.

3. Each reviewer on a team writes a Recognition Report in AIMS prior to team discussion.

4. Lead Reviewer on team will contact team to establish timeline for completing review and date(s) for discussion of findings. Teams use CAEP teleconference system.

5. Lead Reviewer submits final Recognition Report on behalf of the team.

The Basics to Keep in Mind• DATA RULE• Minimum requirement for data are from two applications of

the assessments (per program location)• For revised or response to condition reports, data from one

application of assessments are required• RESPONSE TO CONDITIONS REPORTS• Reviews may address only the conditions itemized in Part G of

the prior recognition report• USE OF GRADES• SPAs must accept grades as an assessment of candidate

knowledge. See the reporting rules on grades at http://www.ncate.org/Accreditation/ProgramReview/GuidelinesAndProcedures/DocumentingCourseGrades/tabid/456/Default.aspx

More Basics…..• WRITING CONDITIONS• Condition statements should be specific so that program has

enough information to make changes. Use phrases from the reviewer report writing document when possible for consistency. AVOID providing suggestions for changes and instead, write descriptively about what is missing, restate what is required, etc.

• OVERALL APPROACH• The review process is not punitive for programs – but rather

collaborative and collegial, using a holistic approach. However, avoid “pity scores.” Going easy on programs in the first review makes subsequent review(s) extremely challenging.

Program Report – Section ISection I provides the background info needed to make sense of the report and to complete portions of the Recognition Report. Two narrative responses plus:• Candidates and Completers chart• Faculty chart• Program of Study

• Read carefully for the following:• Are there sufficient full-time faculty for the program?

• Does the faculty have credentials in gifted education?

• Do the numbers of completers provided here approximate the number in the assessment data charts? (if not, you should include a comment that asks for for an explanation)

• What info can you use to describe program strengths?

• The Section II chart tells you what assessments the program has submitted, and where the assessments take place in the program.

• The Section III chart indicates which assessments provide evidence that candidates have mastered the knowledge and skills for each standard.

As you read these sections• Note the variations in the types of assessments (formative or

summative) • Note the “n” for the data tables; the number taking the

assessments and the number of completers reported should very closely match.

Program Report - Sections II & III

Section IV – the Heart of the Program Report

For each of the assessments submitted, the program must provide:• A narrative including a description of the assessment, the

alignment of the assessment to standards, analysis of the data from the assessment, and an explanation of how the assessment provides evidence of meeting standards; and

• Documentation of the assessment (attachments to the report)• the 2-page narrative• the assessment itself (or directions for completing the

assessment)• the scoring guide, which is organized by standard/element • the data chart that shows one or two applications of the data,

presented in a way that reflects the scoring guide and is disaggregated by performance level

Program Report – Section IV

Section V• The program describes how it has used the data from the

submitted assessments as a way to evaluate and make changes to its program.

• For example, the subscores on a state test may show that

candidates score lower in a particular domain or skill. This should have prompted the program to consider whether it needs to increase attention to that area within the curriculum – or just monitor that area carefully in future data reports.

Program Report – Section V

Section VI: Required only in Revised or Response to Conditions reports.

• This section should tell the reviewer what the program has done to address the concerns or conditions to recognition specified in the previous report, as well as provide a summary of what has been submitted in the current report. Reviewers have access to the previous program report and to the previous recognition report.

Program Report – Section VI

It’s All About the Assessments…. Which Should• Align to standard/elements (course assignment to scoring guide to

data chart to standard)• Assess meaningful knowledge and skills (key aspects of

standard/element are addressed) • Include a scoring guide (rubric) that shows distinct performance

levels (observable and measureable content-specific behaviors that show qualitative differences from one level to the next); and flows from a stem descriptor that includes the elements of the standards

• Include data that show that candidates meet standard • Data chart aligns to each standard/element described in the

rubric/scoring guide. Number and percentage of candidates performing at each level for each standard/element is reported.

Assessment Alignment to the NAGC-CEC Standards: What to Look For• Determine if the key elements contained within each NAGC-CEC

Standard are evaluated within the assessments, as claimed in the program report.• Some assessments will address multiple standards.• Some assessments will claim to assess multiple standards, but

do not. (No penalty for that as long as each standard is covered by at least one dimension within the rubric or checklist.) Make a note in the comment field of the recognition report of the standard(s) not addressed by an assessment.

• Some standards are covered by assessments not claimed in Section III of the report. Give credit and make a note in the comment field for that standard.

Most Common Issues Found By Reviewers - Assessments• Assessments and scoring guides are difficult to interpret

because of missing elements, data.• Reviewers should unravel as much as possible and describe the

situation as specifically as possible in the comment sections so that the program can make improvements for the revised submission.

• Assessments align to standards as a whole but are too general to evaluate candidate mastery of specific concepts contained in the standards.

• Candidate self-assessment, by itself, is not acceptable as a key assessment.• Self-assessments where results are reviewed and discussed with

faculty or supervising teacher can be acceptable assessments.• Assessments offer assignment choices that focus on different

knowledge and skills, which may result in gaps in standards mastery.

Common Issues - Rubrics• Rubric descriptors mirror the language of the standards

rather than describe candidate expectations in operational terms.

• Rubrics/scoring guides are too general and do not describe the expectations for specific concepts from the standards.

• Rubrics are not aligned to the activities described in (or components) of the assessments.

Rubrics, cont’d• Rubrics do not isolate individual standards or elements• It is possible to group a few closely related elements

together in one rubric category. Note: we would expect to see fewer instances of this with the 2013 standards, which have 28 elements.

• Rubric do not describe meaningful differences between levels of mastery on the assessment. • Quantitative rather than Qualitative descriptors: requiring

4 references for something, compared to 2, is not a meaningful difference between meeting/not meeting expectations.

• Subjective qualifiers ("most," "somewhat," "exceptional") are not meaningful ways to describe differences between levels of performance.

Problem Rubric DescriptorsMimics standard languageCandidate provides evidence of differentiated instructional plans that• Align with state and national

academic standards• Are created with the individual

student in mind• Use a variety of curriculum

resources, strategies, and product options for differences among individuals with gifts and talents

• Address academic and career guidance experiences into the learning plan

Reliance on subjective descriptorsThe candidate provided sufficient evidence, or description, of the following: (list of elements within a standard follows)

OR

The candidate demonstrated competency beyond standard expectation levels in understanding instructional planning for the gifted specific to the targeted criteria.

Problem Rubric Descriptors

Expectations are based on concepts in the standard elements but are not described

Competent demonstration of understanding, with satisfactory descriptions and defense of answers.

Evaluation based on qualities not related to the NAGC-CEC standards

Provides significant information about the purpose, organization, and rationale for the selection of documents (for portfolio).

Common Issues - Data• Data Charts are not aligned to the scoring guides. • Data presented as a total score, rather than by each

standards-based concept presented in the rubric/scoring guide. The reason to disaggregate in this way is for the program to be able to spot the strength/weakness areas and make adjustments for program improvement.

• Insufficient data provided. • Data not disaggregated by # of candidates scoring at each

performance level on each item in the scoring guide• Data are inconsistent with numbers of program completers.• Administration date not identified• Data charts do not report separately where program is at

multiple sites• Where grades are submitted, the program must also submit the

program/institution’s minimum grade point/grade requirements.

Evaluating Individual Standards as You Review Each Assessment

Standard/Element is Not Met

Standard/Element is Met with Conditions Standard/Element is Met

Assessments assess meaningful content specific knowledge and skills for the standard.

Assessments fail to measure key components of the standard.

ORAssessments consist of simply a checklist of items to be included in the assessment and do not address the quality of candidate performance.

Numerous items are presented on the assessments; however, the items are limited in scope or only partially provide evidence for meeting of the elements of the standard.

OR The assessments fail to define candidate behavior at each level in operational terms. For example: Levels of candidate proficiency are differentiated in terms of quantity of the same behavior, such as familiarity with 4 key references (vs. 2, vs. 1).

OROne item is purported to align with multiple standards and not one individual standard. While an assessment may provide evidence for multiple standards, individual items on the assessment usually cannot provide adequate evidence for multiple standards.

Assessments identify key components of required content specific knowledge and skills and provide evidence of candidate knowledge and/or attainment of the standard.

Standard/Element is Not Met Standard/Element is Met with Conditions

Standard/Element is Met

Scoring guides (rubrics) assess distinct levels of candidate proficiency

Scoring guide (rubric) consist of a checklist of behaviors that can be answered yes or no. Behaviors are not defined or expectations identified. Performance levels are unclear and/or subjective, potentially allowing for biased results.

ORScoring guides (rubrics) are inconsistent or incomplete.

ORDistinctions between performance levels are not clear.

The assessments fail to define candidate behavior at each level in operational terms.For example: Throughout the scoring guide (rubric), descriptions of candidate proficiency are not objective, using such terms as “consistently”, “occasionally,” “somewhat,” “never” or “exceptional”

ORScoring guides (rubrics) rely on quantity, rather than quality.

Scoring guides (rubrics) identify distinct levels of candidate proficiency in terms of criteria, are content specific, observable, and measurable behaviors, allowing for fair and unbiased results. Moreover, they use a scale with descriptors of each item to be rated. Quality and quantity indicators are employed as appropriate.

Preponderance of evidence

While some evidence is provided, it is insufficient for reviewers to determine the standard is met.

ORAssessments fail to assess the key concepts within the standard.

ORAssessments fail to assess the majority of key components of the standards.Data presented as evidence are comingled; thereby, making it difficult for the reviewer to determine if the standard is met.

Multiple assessments are provided for meeting standards, but provide only partial or marginal evidence. The assessments submitted only partial alignment with the standard.

Sufficient evidence is presented in the required format for reviewers to determine that the key concepts within the standard have been assessed.

Standard/Element is Not Met Standard/Element is Met with Conditions

Standard/Element is Met

Data demonstrate that candidates have met the standard/element

Data charts do not align with the assessment. Data charts fail to identify percentage of candidates at the acceptable level and simply report a mean score. ORA generic scoring guide is used that simply assigns a value across all items. ORData are missing from the chart. Data are reported by individual student and not aggregatedInsufficient data are provided; therefore, the reviewer cannot determine if the standards are met.Insufficient data are presented or data are not disaggregated to the program level.

Program fails to report the overall number of candidates. Data charts do not align directly with the scoring guide. The data charts report mean scores for categories while the scoring guide is organized by item or percentage of candidates achieving a specific level and are reported for the category and not individual item. ORData are aligned to multiple standards and an aggregate score is reported for an overall category. Data must be reported at the same level as it was collected in the assessment. If data are collected on individual items, it must be reported by individual items and not an average or overall score for the assessment or for a category.

Data charts are aligned with the assessment; percentage and/or mean and range of candidates achieving the acceptable level is reported; charts are correctly labeled; and all required data are reported.

The Recognition ReportPart A: Recognition Decision• Complete this section last. • National Recognition under the 2013 standards is more stringent

than under the 2006 standards. See separate slides

Part B: Status of Meeting Each NAGC-CEC Standard• See separate slide

Part C: Evaluation of Program Report Evidence• This section is completed using evidence from assessments

provided, informed by reviewer’s expertise.

Recognition Report, cont’d

Part D: Evaluation of the Use of Assessment Results• Comments in this section derive primarily from information

provided in section I of the program report (context).Part E: Areas for Consideration• Not required to complete. Comments should be over-arching

comments, such as increasing the number of faculty that have expertise in gifted education; ensuring candidates are working with multiple gifted K-12 learners.

Part F: Additional Comments• Not required to complete. Space provided to draw attention

to accreditation examiners. For example, calling attention to the lack of faculty expertise in gifted education.

Recognition Report, Part B

Part B: Status of Meeting Each NAGC-CEC Standard• This is the heart of the report writing. Designate each standard

as met, not met, or met with conditions after reviewing the assessments from a holistic approach (“preponderance of the evidence”) to determine the degree to which candidate mastery of the breadth/depth of the concepts in each standard is assessed and measured.• Comments are helpful; comments are required when decision is

“met with conditions” or “not met.” Not appropriate to include comments directing how the program could/should change an assessment; focus is on explaining what needs improvement.

• See the next slide for the differences between decisions for programs using the 2006 standards vs. those using the 2013 standards.

Individual Standard Decisions 2006 vs. 2013 StandardsFOR THE 2006 Standards• For a standard to be determined as “met,” more than 50% of

the elements (100% is not required) in that standard must be measured and assessed within the assessment(s) scoring rubrics. (See the NAGC Guidance for Program Reviewers for 2006 standards for more details.)

FOR THE 2013 Standards• For a standard to receive a “met” determination, all the

elements of the standard have been addressed and a preponderance of the evidence presented shows that the candidates are meeting the expectations of the standard. (See the NAGC Guidance for Program Reviewers for 2013 Standards for more details.)

Report Writing: Directions for Writing CommentsIn Part B -- please compose each comment in the following way:1. “The program indicates that assessments (#s) address standard (#).”2. Next, address each assessment in sequence; begin each para with

“Assessment (#) (include label for assessment the first time it is referenced)…and continue with brief comments about what concepts/elements of the standard are addressed by the assessment.

• If the assessment does not address the standard as claimed, make a brief note of it.

• If issues with an assessment, scoring guide/rubric, or data, indicate the problem, being as specific as possible. These comments will become part of the conditions statements listed in Part G of the report. Can be as simple as “Assessment 7: The rubric does not include criteria for this standard,” or “Assessment 4, which is a self-report by the candidates, is not acceptable as a key assessment.”

Writing Comments, cont’d

3. The final para of comments for each standard should conclude with a statement such as, “The assessments address all of the elements of standard (#) and the data support candidate mastery; therefore, the standard is determined to be met.”

• As you move through the standards, no need to repeat comments about same issue/problem with an assessment, simply refer back to the first standard where the problem was described: “Assessment 5: see comments under standard 4.”

• Use the Reviewer Report Writing Document for assistance in choosing phrases in describing issues with assessments, scoring guides (rubrics), and data. Using these phrases ensures consistency across SPAs, which is helpful to universities when they receive multiple program reports.

Importance of Clear Conditions

• Conditions describe the concerns about the assessments that show candidate mastery of the NAGC-CEC standards. Conditions guide program improvement and are the focus of a revised or response to conditions report. Thus, it is essential that the conditions statements are as clear as possible.

• Keep in mind that subsequent reviews focus only on the conditions statements developed by the previous reviewers. Please be sure that you are as thorough as possible in describing the concern, taking care not to write directive (“the program should…”) language.

• Using same descriptors for same/similar issues with assessments helps reduce follow up questions / confusion from programs.

• Again, use the Reviewer Report Writing Document for assistance in choosing phrases where possible.

Reviewing Revised or Response to Conditions Reports

• If possible, the report will be assigned to at least one reviewer from the original review.

• If Revised, reviewers will only evaluate standards that were previously not met or met with conditions.

• If Response to Conditions, reviewers only address issues listed in Part G

• Reviewers may not reverse previous decisions on “met” standards or add new concerns or conditions unrelated to concerns/standards addressed in the second report.

Program Recognition Decisions

After completing Part B of the Recognition Report, any conditions statements should be included in Part G.

Lastly, return to Part A to make a recommendation on the program’s recognition status, using the following guidelines, which differ for programs being reviewed under the 2006 vs. 2013 NAGC-CEC standards. See next slides.

Recognition – 2006 StandardsFor Programs Using the 2006 NAGC-CEC Standards

• National Recognition – program meets CAEP’s 80% rule on state licensure exam (where applicable), 8 of the 10 individual NAGC-CEC standards are determined to be “met,” and the program meets the data requirements.

• National Recognition w/Conditions – program received “met” or “met with conditions” on 5 or more of the individual standards.

• Not Recognized – program met or “met with conditions” 1-4 of the individual standards and needs major improvements in most or all areas noted above.

Recognition – 2013 Standards

For Programs Using the 2013 NAGC-CEC Standards

• National Recognition – program meets CAEP’s 80% rule on state licensure exam (where applicable); all 7 of the individual standards must be determined to be “met.” **

• Not Recognized - a program received “not met” on 1 or more individual standards.

• National Recognition with Conditions – applies to all other cases. **

** Where programs have received “met” on 6 standards and “met with conditions” on 1 standard, reviewers should use their professional judgment in considering the conditions for the 7th standard to determine whether national recognition status is appropriate.

Final Steps for Completing Review

• The Lead Reviewer’s task is to summarize team discussion(s) and using individual reports write up the final Recognition Report and post to the CAEP AIMs system.

• Reviewers should check the final report for fairness, bias, and ensure the report is clear and contains sufficient explanation, using the recommended phrasing where possible.

• Note: An NAGC program auditor will review the team report, making any edits for consistency, and submit a final report to CAEP.

CAEP will be moving materials from the old NCATE website to the new site (http://caepnet.org/). • In the meantime, it is still possible to access archived webinars and

other resources at http://www.ncate.org/ProgramReviewers/tabid/104/Default.aspx

• Also, CAEP staff (Elizabeth Vilky), NAGC’s SPA Coordinator (Jane Clarenbach), and your review partners are all resources to help you get comfortable and to answer questions.

Resources for Reviewers

Contact Information

• Contact at CAEP (Elizabeth Vilky, Elizabeth.Vilky@caepnet.org ) if you have problems with technology, accessing the report, contacting a team member, or an emergency that requires you to withdraw from your assignment (or for general questions as well). (202-466-7496)• Contact at NAGC (Jane Clarenbach, janec@nagc.org) if you

have questions about NAGC-CEC standards, alignment of standards in assessments, writing a quality report, etc. (202-785-4268)