Slide  1  

NAEP Questions Toolhttp://nces.ed.gov/nationsreportcard/itmrlsx/

 

Christine  Kempf  

Ohio  Department  of  Education  

NAEP  Administrator  

614-­‐995-­‐0111  

Christine.Kempf@education.ohio.gov  

Slide  2  

NAEP - Quality Assessment ItemsChristine Kempf October, 2013  

 

Slide  3   What is NAEP?• NAEP stands for the National

Assessment of Educational Progress.• “The Nation’s Report Card”• Largest nationally representative

assessment of what America’s students know and can do.

• Only assessment administered nationally in United States

• Considered “Gold Standard”

3

 

 

Slide  4   National Assessment of Educational Progress (NAEP)• 1969: now national and state results• Grades 4, 8, and 12• Subjects: reading, math, science, writing,

US history., geo. civics, etc.• Late January through early March

 

 

Slide  5  NAEP 2014 and Now!

• Grade 8 Science and Technology and Engineering Literacy (TEL) – National only

• Grade 4 and 12 - Science pilots

• Ohio NAEP 2013 results: Grades 4 and 8 Reading and Mathematics Thursday, October 31 11 AM EDT

• Ohio NAEP TIMSS 2011Linking Study Results

 

 

Slide  6   NAEP

 

 

Slide  7  

Multiple  choiceConstructed  Response  types

NAEP Question ToolOver 3000 NAEP released items (from 1990 through 2011, 2013 pending)

Grades  4,  8,  12

 

 

Slide  8  Questions Tool

http://nces.ed.gov/nationsreportcard/itmrlsx/default.aspx

 

 

Slide  9  NAEP Questions Tool-Selection

 

6  Tutorials  (short)    

Overview  and  general  features  Search  Basics  Customizing  search  results  list  Previewing  questions  and  question  details  Workspace  basics:  adding  questions  and  arranging  the  list  Creating  sample  question  documents  and  data  reports    

Slide  10  Question Selection

 

“What  can  I  do  here?”    on  each  page  and  explains  functions  per  screen  or  section.    

Slide  11  Refine Search

Grade

Item Type

Difficulty Levels

 

Deselect the criteria you do not want to use in your search. May need to use the scroll bar. Select Content Classifications to narrow your search  

 

Slide  12  

Medium questions  are  those  that  40  to  60  percent  of  students  answer  correctly

Difficulty LevelsEasy questions  are  those  that  over  60  percent  of  the  students  answer  correctly

Hard questions  are  those  that  fewer  than  40  percent  of  students    answer  correctly

 

National  Public  is  data  from  public  schools  in  the  nation.    National  is  data  from  public  and  private  schools  in  the  nation.    

Slide  13   Refine SearchContent Complexity Year Keyword

 

For Reading -New vocabulary questions are now available in the NAEP Questions Tool. Deselect the criteria you do not want to use in your search. May need to use the scroll bar. Select Content Classifications and Years to narrow your search    

Slide  14   Refine SearchContent Complexity Year Keyword

 

For Reading -New vocabulary questions are now available in the NAEP Questions Tool.

Deselect the criteria you do not want to use in your search. May need to use the scroll bar. Select Content Classifications and Years to narrow your search    

Slide  15  Customize Item Information

 

Open  Show/Hide  column  header  to  reorder:  Check  columns  for  item  details  desired.  There  are  more  column  headings  for  item  characteristics  as  you  scroll  down  on  the  Show/Hide  arrow.    For  example,  you  may  uncheck  Block  column  and  #  column  and  add  Complexity  to  main  view  next  to  the  Difficulty  column.    You  can  also  sort  columns  with  arrows  next  to  the  column  heading.    

Slide  16  Question Details

 

All  needed  artifacts  and  materials  come  with  an  item.    

Slide  17  NATIONAL DATA:Pub & NonPub

 

National  Public  is  data  from  public  schools  in  the  nation.    National  is  data  from  public  and  private  schools  in  the  nation    

Slide  18  State Data

 

 

Slide  19   OH Student Response Data

 

Omitted  Response:  Have  no  response  but  are  followed  by  at  least  one  response  to  a  later  item;  this  is  considered  an  intentional  behavior,  in  contrast  to  an  item  that  was  not  reached  because  of  time  limitations.    

Missing:  associated  with  items  at  the  end  of  each  block  that  students  did  not  have  an  opportunity  to  respond    

Off-­‐task  response:    (Constructed  response)  A  response  that  is  unrelated  to  the  question  being  posed;  differs  from  an  incorrect  response  to  the  question  or  an  omitted  response.    

Slide  20  Constructed Response

 

Slide  21  Constructed Response

 

 

Slide  22   Constructed Response

 

 

Slide  23  Selecting Items

 

 

Slide  24   Build an Assessment

 

Click  on  My  Workspace  to  see  your  selected  items.  My  Work  –Create  a  Document  is  on  the  left.    If  a  question  has  a  needed  item  such  as  a  ruler  or  triangle,  etc.,  it  will  be  printed  with  the  question.    

Slide  25  

 

Select  Data  such  as  National  Totals  and  Total  for  Ohio  (Include  Nation  will  be  National  Public  which  is  public  schools)  Select  Format    If you get a “pop-up” blocker on your computer you may see this message. Just click where it says “click here” and then choose “download file” If more than one item was selected, different tabs will display the stats for each item. National  Public  is  data  from  public  schools  only  in  the  nation  (is  displayed  with  state  total  when  “include  nation”  is  selected).  National  is  data  from  public  and  private  schools  in  the  nation    

Slide  26   Interactive Computer TasksHands-On Tasks

http://nationsreportcard.gov/science_2009/ict_summary.asp

 

 

Slide  27  International Assessments

 

If  the  NCES  site  is  not  available,  you  can  use  the  international  assessments  own  websites  for  accessing  the  items  and  information.    

Slide  28  

Other States: Michigan, Iowa, NC

Source for ItemsNew York City:keyword Test Items / tests/ NAEP http://schools.nyc.gov/default.htm

www.allthingsassessment.info/

NAEP sample test booklets http://nces.ed.gov/nationsreportcard/about/booklets.asp

 

Samples  are  given  on  the  sheet  for  each  grade  from  a  different  source  but  the  website  contains  items  for  other  grades  also.    

Slide  29   TIMSS• Measures student learning in mathematics

and science at grades 4 and 8

• Compares achievement of students within about 50-65 countries/educational systems (2011= 38 countries, 9 education systems)

• 2011 results: http://timss.bc.edu/index.html

• 2011NAEP-TIMSS Linking results (Ohio) http://nces.ed.gov/nationsreportcard/studies/naep_timss/

 

Trends  in  International  Mathematics  and  Science  Study    TIMSS    

Slide  30   TIMSS Math 2011–Linking Results:• Korea,Republic (613)    • Singapore  (611)• Chinese  Taipei-­‐CHN  (609)  • Hong  Kong  SAR1  (586)• Japan  (570)• Massachusetts  (561)  • Vermont  (547)  • Minnesota  (545)     New  Jersey  (545)• New  Hampshire  (541)• Russian  Federation  (539)• North  Carolina  (537)• Maine  (535)  Wisconsin  (535)  • Quebec-­‐CAN  (532)  • Montana  (531)  • New  York  (529)• North  Dakota  (528)• Pennsylvania  (527)  Iowa  (527)  • Alaska  (524)  Wyoming  (524) Kansas  (524)  • Virginia  (523)  Washington  (523)  Ohio  (523)  • Indiana  (522)• Rhode  Island  (521)  South  Dakota  (521)  • Colorado    (518) Connecticut    (518)  • Illinois  (517)  • Israel  (516)  • Texas  (515)   Delaware  (515)  • Maryland  (514)  Finland  (514)  • Florida  (513)    Oregon  (513)    Idaho  (513)

 

2011  NAEP  –TIMSS  Linking  Study  :State  Math  Performance  Thirty-­‐six  states  scored  higher  than  the  international  TIMSS  average  (500)  in  mathematics.    Ten  states  had  scores  that  were  not  significantly  different  from  the  average  and  6  states  had  lower  scores.      

Slide  31  Benchmarking – TIMSS 2011 Math

Massachusetts,US 561 (5.3) hMinnesota,   US 545 (4.6) hNorth Carolina, US 537 (6.8) hQuebec, Canada 532 (2.3) h

Indiana, US 522 (5.1) hColorado, US 518 (4.9) hConnecticut,   US 518 (4.8) hFlorida, US           513 (6.4)               hOntario, Canada 512 (2.5) hAlberta, Canada 505 (2.6)

California, US 493 (4.9)

Dubai, UAE 478 (2.1) iAlabama, US 466 (5.9) iAbu  Dhabi, UAE 449 (3.7) i

Benchmarking Participants

 

 

Slide  32  TIMSS Gr. 8 Science Item

 

 

Slide  33   2011 TIMSS Science 8

 

Content  Domain:    Physics    Topic  Area:    Energy  Transformations,  Heat  and  Temperature    Cognitive  Domain:    Applying    Maximum  Points:  1    S06_13  (S042407):    ice  block  CR  (1  Point)  Country        Percent    Correct  Chinese  Taipei        60      (2.2)        h  Singapore                    53      (2.2)        h  Japan                                51      (2.0)        h  Ukraine49  (2.9)  h  Chile  45  (1.9)  h  Tunisia  43  (2.2)  h  Korea,  Rep.  43  (2.0)  h    etc.            

Slide  34   S06_13 (S042407): Which ice block will melt firstBenchmarking Participants 2011 TIMSS Gr 8

Alberta, Canada 59 (2.2) h

Ontario, Canada 49 (1.9) h

Quebec, Canada 46 (2.3) h

Minnesota, US 36 (2.2)

Massachusetts, US 34 (3.1)Florida, US 34 (3.7)Abu Dhabi, UAE 31 (2.3)

Colorado, US 31 (3.1)Dubai, UAE 30 (2.4) i

California, US 28 (3.2) i

North Carolina, US 28 (3.0) i

Connecticut, US 28 (2.3) i

Indiana, US 28 (3.1) i

Alabama, US 20 (2.0) i

 

 

Slide  35  TIMSS 2011 Released Items

• Released items: characteristics chart, item, key• Released Items Statistics - pdf & excel:

countries/education systems rank and percent correct

• http://timssandpirls.bc.edu/timss2011/international-released-items.html

To view or download one or more of the released item sets, please use the links below. Grade 4 Released Items and Percent Correct Statistics:Mathematics ScienceGrade 8 Released Items and Percent Correct Statistics:Mathematics Science

 

 

Slide  36  PIRLS

• Measures students’ reading comprehension of literary and informational text at grade 4 every 5 years since 2001

• Compares achievement of participating countries/educational systems students (about 55 countries/educational systems)

• Examples of released PIRLS items can be viewed at http://timssandpirls.bc.edu/pirls2011/index.html

• Results: http://timssandpirls.bc.edu/pirls2011/international-results-pirls.html

 

Progress  in  International  Reading  Literacy  Study    

Slide  37  PIRLS Gr. 4 Reading Item

 

 

Slide  38  PIRLS

 

 

Slide  39   PISA:Programme for International Student Assessment

• 15 year old students

• Reading, mathematics and scientific literacy

• Every 3 years with one subject assessed in depth (Math 2012)

• About 70 countries and economies• PISA: http://www.oecd.org/pisa/• Released Items and results:

http://www.oecd.org/pisa/pisaproducts/

 

 

Slide  40   PISA Math Item: Speed of a Racing Car

What  is  the  approximate  distance  from  the  starting  line  to  the  beginning  of  the  longest  straight  section  of  the  track?

 

 

Slide  41  Science InteractiveGrade 4 Science Challenge

20 Items, computer-basedDeveloped through US DOE grant/NSFAVAD Access-Enhanced Science Items:• http://ed.sc.gov/agency/ac/Assessment/Acc

ess-EnhancedScienceItems.cfm

 

 

Slide  42   Uses with TeachersProfessional Development

Correlation to CCSS or OH SLOItem development (distractors)Scoring Training – Constructed ResponseUse state and national performance data

Assessment DevelopmentFormative – Summative -- Benchmark

 

 

Slide  43  Uses with Students

Improve students abilities for self-assessment

Teach higher reasoning skills/strategies

Diagnostic information

Test taking strategies

Use state performance data to focus instruction

 

 

Slide  44  Copyright Law

• The questions in the NAEP Questions Tool and other resources may be reproduced as long as any copyright restrictions are observed for third-party copyrighted material.

• If you publish any part of NAEP questions, include the acknowledgment below. The year and the name of the assessment you are using should appear at the end of the statement for any question source (TIMSS, PIRLS, PISA).

– SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), year, name of the assessment.

 

Christine  Kempf  

Ohio  NAEP  Administrator  

614-­‐995-­‐0111  

Christine.Kempf@education.ohio.gov  

  education.ohio.gov

 

Facebook      Ohio  Families  and  Education  

                                       Ohio  Teachers’  Homeroom  

Linked  in          ohio-­‐department-­‐of-­‐education  

Storify                  storify.com/ohioEdDept  

Twitter                @OHEducation  

YouTube          OhioEdDept