NSTA Program Renewal online workshop. Picture from: http ...static.nsta.org/pdfs/NCATE-ReviewerRenewalFall2007.pdf• Consider your role as a reviewer Remember – most programs are

1

NSTA Program Renewal online workshop.Picture from: http://www.adirondackartistsguild.com

2

NSTA Reviewer Renewal Online Training

September 19, 2007Erica M. Brownstein

Capital University

NSTA Preservice Accreditation Program Coordinator

For information contact: [email protected]

This training is focused on reviewers that have not reviewed for a year or more. The alignment between the NCATE system and the NSTA standards has been refined and clarified.The next session, October 12 at 1:00 EASTERN time, will focus on feedback to programs.

3

Objectives

• See the relationship between NSTA 2003 Standards and the NCATE Assessment System

• Consider your role as a reviewer

Remember – most programs are working very hard to make good science teachers. Be constructive and positive. We are here to help them communicate what they are doing. Many times, it is a matter of communication – we can only see the words that they send.

4

Agenda

Goals for workshop

• NSTA/NCATE alignment

• Changes in the alignment

• Each assessment highlighted

• Questions by attendees

For some of you, parts of this you will have heard before. But there are parts that no one has heard, so I hope you will benefit from the time spent.

5

Introductions

• Name• Institution• How does becoming a reviewer fit with

your professional goals? *PUT ON CARD*• What new thing have you learned about

your program in the last year?

6

Tensegrity

Balance the NSTA standards and the uniqueness to each program

7

NCATE Assessments

• Maximum of 8 assessments• Assessments 1 through 5 are the same for all

SPA’s• Assessments fall into 3 categories

– Content– Pedagogy– K-12 Student Learning

8

Attachments

• NSTA / NCATE program alignment chart

• NSTA matrix – One line description of each standard– Underneath each required assessment

Have attachments near you to write on and refer to

9

PreviousNSTA / NCATE Alignment

10. Professional Dev.9. Safety & Welfare8. Assessment 7. Science in the Community6. Curriculum5. Skills of Teaching4. Issues in Science3. Inquiry2. Nature of Science

Math in scienceResearch in scienceTechnology in Sc.Unifying ConceptsConcepts & Principles

87654321Standards1.

Sc.

con

tent

If you are familiar with this chart, note the areas that have changed since June of 2007.We have eliminated areas that are common to all programs – to help eliminate redundancy AND let Science Education report focus on items that are unique to science.

10

PresentNSTA / NCATE Alignment

9. Safety & Welfare8. Assessment 7. Science in the Community6. Curriculum4. Issues in Science3. Inquiry2. Nature of Science


87654321Standards1.

Sc.

con

tent

Look at your attachment chart – It should look like this one. This is my B&B when I am reviewing. I keep a printed copy and write on it as I am reviewing.If you are familiar with this chart, note the areas that have changed since June of 2007.We have eliminated areas that are common to all programs – to help eliminate redundancy AND let Science Education report focus on items that are unique to science.

11

1 ContentPraxis II or State Test



87654321Standards1.

Sc.

con

tent

Refer to your chart – Praxis II or the State Test results are for Standard 1a ONLY. But also note that 1a has other assessments that are required, so the standard cannot be “met” through assessment 1 only.It must also be met in Assessments 2, 3, 5 and 8.

12

1 ContentPraxis II or State Past 3 years completers

• Must include all completers• 80% pass rate applies if 10 or more total (all

science content tests for all 3 years), • Data to include

– Mean scores– Range scores– Pass rates for state– Sub-scores for latest year– Disaggregated by program

• Do not need to show content alignment if Praxis II

All data should be shown together. There also needs to be some disaggregation of the data.Disaggregated by program – so the Undergraduate data, post degree, graduate data should be separate. Also, data should be disaggregated by licensure (this often is just by taking separate tests).NCATE requirements are the 80% rule and the content alignment of the standardized test. If they forget to include the State Test alignment with 2003 NSTA Standards – just make a note of it. Only request it if they are met with conditions or not met. Do make sure the test is science content (not pedagogy or a general skills test).

13

Pause for thought1 Content: Praxis II or State Test

• How much data is required?• What if it is a State test?• What if there are less than 10 completers in

3 years?– What about different licensure areas or levels of

degree?• What about sub-scores?• What comments should be made?

Keep in mind that it is ONE measure of content. NSTA has 3 measures for science content. IHEs live with the tests they are required to give. They also have limitations on the data they can acquire. Make comments of absent information or data. But also recognize any limitations that have been noted.

14

2 Content Science GPA & Science Alignment



87654321Standards1.

Sc.

con

tent

This assessment has two measures of science content – GPA and Alignment Chart

15

2 ContentGPA and Science Content

There are two components: – GPA– Science Content Analysis Forms

GPA• Grade distributions or GPAs must be

disaggregated• Provide grade data from all science courses

taken by candidates in the program.– May be all science courses or all science courses

within the major field of licensure

Disaggregated by:licensure areas of the candidates (i.e., biology, general science, etc)distinct levels of licensure (middle level, high school, secondary, etcdegree levels of preparation (undergraduate/post-baccalaureate/ masters)

16

2: Content Analysis Forms• Available on the NSTA and the NCATE website• Science content coursework or proficiencies must

align with the NSTA 2003 Standards• Must demonstrate a 90% alignment

– In each discipline cluster (core, advanced, supporting)• Compare the submitted form with the original

– # of cells• Compare the coursework submitted with the required

courses. Note requirements for all areas– Undergraduate– Post degree– Master’s

• Note any areas that would not be expected (an introductory physics course aligning with the ecology concepts).

Post degree and masters must also demonstrate that they advise their candidates to meet the content requirements. For example, a major in physics will not likely meet the supporting competencies needed for a physics licensure.

17

3 PedagogyUnit Plan



87654321Standards1.

Sc.

con

tent

Must be a unit Must include minimum levels of performance for each of the NSTA standardsDemonstrates pedagogy only – not evidence of student learning nor science content.

18

3 Pedagogy: Unit PlanAt least one major teaching unit

(not just a single lesson plan) that includes requirements for activities with lesson plans and various

assessments. The unit must include:Often includes:– Science concepts &

principles (1a)– Technology in Sc. (1c)– Curriculum (6)

• National and State– Assessment (8)

Must also include:– Unifying Concepts (1b)– Nature of Science (2c)– Inquiry (3b)– Issues in Science (4b)– Sc. in the Community (7b)

Must be SCIENCE specificMust meet all relevant NSTA standards

The left column are often seen in the Unit PlanPrograms often miss the items on the right:

19

Pause3 Pedagogy: Unit Plan

• What might this look like in programs?

• What if there is an generic Unit Plan –will any of the dimensions be met?

• What if the scoring guide does not match the description given?

Generic – usually will meet Standard 8, sometimes Standard 6.It needs to be very clear that candidates cannot pass the unit unless all areas of science are addressed

20

4 Pedagogy: Student Teaching Form



87654321Standards1.

Sc.

con

tent

Note only safety – all other areas are general to all teachers and the Unit.Remember, safety is essential to a science teacher and be from the science teacher’s perspective. Safety must be present in an observation of a k-12 setting.

21

4 Pedagogy:Student Teaching Form

• May be an addendum to regular form

• Include science specific areas– Safety

• Measurable scoring guides (for safety only)

• Instructions to students, cooperating teachers and university supervisors are included

This is an area where we will accept scoring guides that are not clear (such as Likert Scales that use the standard as a guide). NOTE the problems, but it should not prevent an IHE from a “met” level.Questions? If this is not present, Safety Standard is Not met and program can only be Not MetSafety is more important than learning.

22



87654321Standards1.

Sc.

con

tent

5: Evidence of K-12 Student Learning

CANNOT be a generic instrument.CANNOT be an observation formMust show evidence of K-12 student learning in the scoring guide.Must explicitly address these areas in science

23

5: Evidence of K-12 Student Learning

The most important component of the NSTA program report– Science specific and address each

applicable NSTA standard– Descriptors that are measurable

• Scoring guide – CANNOT be generic– CANNOT be an observation form

Minimum acceptable levels of performance are a Basic or above.

TWS is a fantastic tool to use to evaluate evidence of student learning. However, a generic TWS will not meet unless has an addendum.

24

6 Pedagogy: Safety & Welfare



87654321Standards1.

Sc.

con

tent

Again – must be met to have a program be recognized or be conditionallyrecognized

25

6 Pedagogy: Safety & Welfare

• Focus on what future science TEACHERS need to know and be able to do

• Each area of safety, 9a, 9b, 9c, and 9d, must be assessed at minimum levels

• Generally not met through science coursework or labs

Focus on measurable (reliable/reliable)Met with a large variety of ways (module, test, portfolio, etc.).

26

7 Content: Research in Science



87654321Standards1.

Sc.

con

tent

Research in Science – can be any discipline.

27

7 Content: Research in Science

• Usually performed in the science content courses– It is not “how to do science” – but is a research in a science

discipline– Could be a short project that is embedded in a science

course• A system will need to be in place that

addresses candidates that are post degree or transfer in most science courses

• Action Research or science fair is not acceptable– What would your science departments deem to be

acceptable?

Can be in any science disciplineMust show all aspects – design, conduct, report AND evaluate.

28

8 Content:Contextual Content



87654321Standards1.

Sc.

con

tent

29

8 Content:Contextual Content

• Content understanding of topics important to science educators

• Not generally in lesson plans, unit plans, or observation forms

• This is the most challenging assessment for IHEs.

• Just make sure– It is science content– Each area is addressed with minimum levels of

performance

Has a lot packed into this assessmentBe lenient – but everything must be present(for example, not just an assessment on Issues in Science)

30

Super Six Pitfalls for Science Program Reports

• Not science specific for Unit Plan• Student teaching evaluation form does not have

safety explicit to the teaching of science• Evidence of K-12 student learning is not science

specific• Forgot to include the Content Analysis Forms• Does not have an assessment explicit for safety

content needed for the teaching of science. • Scoring guide descriptors are not measurable

These are the most common areas where program reports fall short of meeting the standards.

31

When examining assessmentsDoes it meet the 5 things to keep in mind for scoring guides?

1. Do the criteria adequately cover the important defining elementsof the desired performance?

2. Do the criteria distinguish the different levels of performance in observable ways?

3. Are the variables in the criteria likely to be interpreted in the same way by different evaluators?

4. Are the criteria just a rehash of the standard or do they definethe standard in operational terms?

5. Is there enough substance to the assessment to assure external reviewers that if the criteria are met, the candidate is prepared to teach to the standards?

#1 problem for all SPA’sPlease use these to help IHEs with their scoring guides. Many are doing the best that they can, but outside input that is constructive may help them in the long run.Remember – it is about making great science teachers.

32

Life of an NSTA program

• Programs written• Submitted to NCATE• NCATE

– Posts– chooses review teams

• Lead and 1 or 2 others– Sends you an email

• You– Contact other reviewer– Read and review report– Email to other reviewer &

post to NCATE• Team

– Discuss the report• Email or conference call• Submit final report

– Audit team• Audits reviews• Final audit sent to NCATE

– NCATE• Edits and sends to

institutions

33

It is most important to communicate with other team members!! Doing that early on in the process and scheduling specific days/times will help everyone.

34

Problems?

I am available for questions. It is my personal mission to help in any way I can.

35

Standards

• Met– All required elements for each assessment– Minimum levels of performance established– Include rationale explicitly for each standard– Include any areas that could be improved

• Not met– Some or none of the required areas included– Include rationale

More details on comments will be given at the October 12 workshopNOTE: Still review Standards 5 and 10 – almost always met – give positive feedback here. It is a constructive relationship with the IHE.

36

Entire Program

• Met– Nationally recognized until next review– Meets all NSTA standards

• Met with Conditions– Meets safety for Assessment 6 & 4– Note ALL items to be changed in special box

• Not met– Does not meet safety

• When in doubt – go for “not met” and the audit team will examine

We have decided to be constructive rather than punitive for the process. That gives IHEs the time and the space to allow their programs to move toward NSTA alignment over the 18 month conditional period.

Documents

NSTA Program Renewal online workshop. Picture from: http ...static.nsta.org/pdfs/NCATE-ReviewerRenewalFall2007.pdf• Consider your role as a reviewer Remember – most programs are