New England Common Assessment Program

August 2007 Science Bias & Sensitivity Review 1

New England Common Assessment Program

NECAP Science

Bias & Sensitivity ReviewWith Reading Passage Review

Supplement

August 14-16, 2007

Killington, VT


State Departments of Education Staff:• Michael Hock – Vermont• Deb Fleurant – New Hampshire• Heather Heineke Agnew – Rhode Island

Measured Progress Staff:• Harold Stephens – NECAP Program Director• Elliot Scharff – NECAP Science Program

Manager• Amanda Smith – NECAP Program Manager• Josh Evans – NECAP Program Manager

Welcome and Introductions


• Committee Member Expense Reimbursement Form

• NECAP Nondisclosure Form• Bias & Sensitivity Guidelines for NECAP item

development

Logistics


What Is Item Bias?

Bias is the presence of some characteristic of an assessment item that results in the differential performance of two individuals of the same ability but from different student subgroups.

Bias is not the same thing as stereotyping (but we don’t want stereotypes in NECAP either).

We need to ensure that ALL students have an equal opportunity to demonstrate their knowledge and skills.


The ability to answer an item correctly should reflect a student’s learning opportunities, not cultural opportunities or life experiences.

No student should be disadvantaged by item content that is insensitive or disrespectful to the student's ethnic, religious, or cultural background.

Think of "culture" in broad enough terms to include disability, language proficiency, socio-economic status, and regional differences.

Item insensitivity is not the same thing as bias, but it is a potential source of bias that should be identified and eliminated.

What is the purpose of sensitivity review?


The “Big Idea” of Bias-Sensitivity Review

NECAP should measure the results of learning opportunities NOT life experiences


Item DevelopmentBias-Sensitivity Review Item ReviewPilot-Testing FeedbackPilot-Testing Data Analysis (DIF)

How Do We Prevent Item Bias?


Review assessment items Complete independent ratings of each

item and record comments Be prepared to discuss “Mixed

Reviews” and “Problem Items” with your review team; Finalize independent ratings

Please Note: The Bias-Sensitivity Review Committee plays an advisory role. Final decisions on which passages and items to

include in NECAP will be made by the NH, RI and VT DOEs.

Role of the Bias-Sensitivity Review Committee


Sensitivity to different cultures, religions, ethnic and socio-economic groups, and disabilities

Balanced gender roles that are free of gender stereotypes Use of positive language, situations, and images Use of context, situations, images, and examples that are

universally understood across groups and cultures Overall, that the test is free of items and text that may elicit

strong emotions in specific groups of students, and as a result prevent those groups of students from accurately demonstrating their skills and knowledge

Overall, that the test is free of items that measure life experiences instead of learning experiences


The Bias-Sensitivity Review Committee DOES need to make recommendations that will help ensure…


Reading Level Grade Level Appropriateness GSE Alignment Language Structure and Complexity Instructional Relevance Accessibility Overall Item Design


The Bias-Sensitivity Review Committee DOES NOT need to make recommendations concerning…

The Item Review Committees Are

Addressing These Issues and Factors.


Process: Day 1 – (1) Training, (2) Practice, (3) Clarification of

Guidelines and Procedures, (4) The entire committee will review grade 8 science, (5) Break into two groups for independent review of item sets—Group 1 will review science for grade 4 and Group 2 will review science for grade 11

Day 2 – (1) Break into small groups for re-evaluation and discussion of “Problem Items” (2) Finalize Independent Ratings, (3) Evaluate the Review Process and Provide Feedback

Day 3 – (1) Training on reading passage review (2) The entire committee will review grade 11 reading passages (3) Break into two groups for independent review of reading passages—Group 1 will review grades 4, 6, and 8 and Group 2 will review grades 3, 5, and 7

The Bias-Sensitivity Review Process


Avoid issues and topics considered sensitive by the general public.

Keep in mind that some topics that are appropriate for classroom discussion may not be appropriate for assessment.

Consider context, setting, and instructional relevance; Bias-Sensitivity Review is not a “literal” process.

Think of “bias” as an assessment issue as well as a cultural issue, that is, a specific topic or feature of a passage or item that may disadvantage one of two otherwise equally matched groups of students.

General Guidelines


Abortion Birth control Child abuse or

neglect Creationism Divorce Incest Occult/Witchcraft

RapeReligious doctrineSexual orientationSuicideQuestioning/

ignoring parent/teacher authority

Topics to Avoid in Science Test Items


Death Impact of humans

on the environment

Natural disasters Conservation,

logging, fisheries Extinction Climate change,

global warming

Sexuality, pregnancy, reproduction, sexually transmitted disease

ViolenceHuman or animal

research subjects, animal rights

PollutionObesity

Science Topics that Need Careful Consideration


Awareness Issues

Avoid economic, regional, cultural, or gender bias in questions, prompts, identified situations, graphics, or reading selections.

Avoid profanity unless it is used in a relevant direct quote.

Ensure that items and passages emphasize academic programs and learning rather than life experiences.

Remember that we are proud of our states and region and expect to see that pride reflected in assessment passages and items.

Remember that items should reflect scientific thinking, processes, and content and not political or personal opinions. Nor should items require students to present political or personal opinions on scientific issues.


New England Common Assessment ProgramBias & Sensitivity Review Committee

Item Review Evaluation Form

Read the statement below and select the rating that best describes your evaluation of the item for potential bias and sensitivity concerns.

“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”

Rating Description

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Comments

Please provide specific comments to explain your evaluation. Your comments should address your particular bias/sensitivity concerns, the subgroup(s) that you feel may be

impacted, and the extent of the potential impact within the subgroup(s).

Item Review Rating Form


Bias-Sensitivity Examples and Non-Examples


Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

2 In a model of our solar system, a tennis ball is used to represent Earth. A marble would be a good representation of Mercury in this model because compared to Earth, Mercury is —A brighterB lighterC smallerD more solid




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree




Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

If a woodland is cut down to make room for a large shopping mall, which of the following would be the most likely outcome for birds living in the area?

a. They will build nests on rooftops

b. They will move to a new woodland habitat

c. They will add garbage to their diet

d. They will stop mating


Tim Kurtz Director of AssessmentNH Department of Education(603) 271-3846

Mary Ann Snider Director of Assessment and Accountability

Rhode Island Department of Elementary and

Secondary Education(401) 222-8492

Michael Hock Director of Educational AssessmentVermont Department of

Education(802) 828-3115

Information, Questions and Comments


New England Common Assessment Program

Bias & Sensitivity Review Reading

Passage Supplement


The “Big Idea” of Bias-Sensitivity Review

NECAP should measure the results of learning opportunities NOT life experiences


Reading Passages are Selected by Item Developers (who are trained to recognize potential sources of bias)

Sensitivity Review of Reading Passages Reading and Math Items are Developed Bias-Sensitivity Review of Reading & Math

Items Item Review of Reading and Math Items Field-Testing Feedback Pilot-Testing Data Analysis (DIF)

How Do We Prevent Item Bias?


Abortion Birth Control Child Abuse/Neglect Creationism Divorce Incest Occult/Witchcraft Questioning

Parental Authority

Rape Religious Doctrine Sex/Sexuality Sexual Orientation Sexually Transmitted

Diseases Suicide Obesity

Topics to Avoid in Reading Passages & Test Items


Death Drugs, Alcohol &

Tobacco Family Issues Guns / Gun Control Handicapping

Conditions Holidays Homelessness

Misuse of Animals/ Animal Rights

Murder Pregnancy Racism, Sexism &

Ageism Religion (Except in

Historical Context) Violence

Topics Which May or May Not Be Acceptable


Passage Review Rating FormNew England Common Assessment Program

Bias/Sensitivity Committee Meeting Passage Review

Grad Level:

Passage # Title

Passage Evaluation

Read the statement below and select the rating that best describes your evaluation of the passage for potential bias and sensitivity concerns.

“This passage does not raise bias and/or sensitivity concerns that will interfere with the performance of a group of students.” Rating Description

4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree

Comments/Explanation

Please provide specific comments to explain your evaluation. Your comments should address your particular bias/sensitivity concerns, the subgroup(s) that you feel may be impacted, and the extent of the potential impact within the subgroup(s)

“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”

Grade Level:


My sister met Kyle at a high school dance and nine months later I was an uncle.

When dad brought the broken-down mare home from the livestock auction, we all thought he had thrown his money away. Now there’s a handsome colt in the next stall, so I guess the old horse was a good deal after all.


“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


What’s the Point?

“Topics to Avoid: Sex/Sexuality”

Not okay because thepassage alludes to

extra-marital sex betweenhigh school students

Okay because our generalpublic standard generally

allows for the notion ofanimal parents and babies

My sister met Kyle at a high school dance and nine months later I was an uncle.

When dad brought the broken-down mare home from the livestock auction, we all thought he had thrown his money away. Now there’s a handsome colt in the next stall, so I guess the old horse was a good deal after all.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


The Christmas Tree Fern is a common sight in the New England woodlands.

Christmas is the most wonderful time of the year.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


What’s the Point?

“Topics Which May or May Not Be Acceptable: Holidays, Religion”

Okay because “Christmas” refers to the name of the

plant; Not a specific reference to religion

Not okay because thereference is based on the

writer’s cultural and religiousvalues (not shared by all)

The Christmas Tree Fern is a common sight in the New England woodlands.

Christmas is the most wonderful time of the year.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


Once, I got my head stuck in our porch railing. If that man with a cell phone hadn’t stopped and called the fire department, I’d probably still be there. Cell phones can save lives.

Everybody at school has a cell phone. Once, during an English test, somebody actually got a call and in a regular speaking voice, said, “Sorry, I can’t talk now because I’m taking an English test.”



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


What’s the Point?

“Avoid ECONOMIC, regional, CULTURAL or gender bias in questions, prompts, identified situations…”

Okay because the writer’sculture and economic status

are not factors in the passage

Not okay because the passageassumes a cultural and

economic status that won’tbe shared by all test-takers

Once, I got my head stuck in our porch railing. If that man with a cell phone hadn’t stopped and called the fire department, I’d probably still be there. Cell phones can save lives.

Everybody at school has a cell phone. Once, during an English test, somebody actually got a call and in a regular speaking voice, said, “Sorry, I can’t talk now because I’m taking an English test.”



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


It was ten years after Uncle Ned’s death when we found his old sea trunk in the attic.

Mary cried and cried when she was told Uncle Ned had died.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


What’s the Point?

“Topics Which May or May Not Be Acceptable: Death”

Okay because the deathreference is used to set the

passage’s time and place but is not a key plot feature

Not okay because death isa key plot feature; deals

with emotions that may beupsetting to some test-takers

It was ten years after Uncle Ned’s death when we found his old sea trunk in the attic.

Mary cried and cried when she was told Uncle Ned had died.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


Pa and Uncle Jimmy drove the final nails into the new barn roof while Ma and I laid out supper. For settlers on the Great Plains in the 1840’s, a barn raising was hard work, but also a time for celebration

Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


What’s the Point?

“Avoid economic, regional, cultural or GENDER BIAS in questions, prompts, identified situations…”

Okay because the genderspecific roles in the passageare historically accurate and

do not, per se, represent bias

Not okay because thewriter’s perspective

represents a clear and unacceptable gender bias

Pa and Uncle J immy drove the final nails into the new barn roof while Ma and I laid out supper. For settlers on the Great Plains in the 1840’s, a barn raising was hard work, but also a time for celebration

Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


“Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.” That’s what I used to think until the day I spied Mary Jean McCubbins up in a tree, building the coolest tree fort I ever saw.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree


What’s the Point?

Okay because the writeridentifies the gender bias inprior beliefs and attempts to correct the stereotype

“Avoid economic, regional, cultural or GENDER BIAS in questions, prompts, identified situations…”

“Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.”That’s what I used to think until the day I spied Mary J ean McCubbins up in a tree, building the coolest tree fort I ever saw.



Rating Descriptions

4 Strongly Agree

3 Agree

2 Disagree

1 Strongly Disagree

Documents

New England Common Assessment Program