FAIR FS Preview webinar 5/16/2014 - Florida State University · FAIR‐FS Preview webinar 5/16/2014 4 4 Teacher: Are my students making progress? How do I differentiate instruction?

FAIR‐FS Preview webinar 5/16/2014

1

Florida Assessments for Instructionin Reading, Aligned to theFlorida Standardsin English/Language Arts

Florida Assessments for Instructionin Reading, Aligned to theFlorida Standardsin English/Language Arts

FAIR–FS

Grades K - 2

Grades 3 – 12


2

OutlineOutline

Setting the context for assessment

Development of FAIR-FS and advantages of computer-adaptive testing

How to administer the FAIR-FS Grades K – 2 tasks Grades 3-12 tasks

Score types and interpreting scores

Instructional implications

Case study – putting it all together


3

Assessment Assessment


4

4

Teacher: Are my students making progress? How do I differentiate instruction?

Parent: Is this the best school for my child?

Student: Am I accomplishing what my teacher expects? How am I doing compared to other students? Administrator:

Are a majority of our students succeeding with our curriculum?


5

Questions Addressed by FAIR-FSQuestions Addressed by FAIR-FS

Which important reading skills are strengths and weaknesses for Stanley?

What skills should be targeted for instruction in order to improve Stanley’s reading comprehension?

What is the likelihood that Stanley will pass the end of year test?

Has Stanley made progress since the beginning of the school year?

End of year outcome assessment:– Is Stanley

meeting grade level expectations?


6

The FAIR-FS is a comprehensive system designed to:

The FAIR-FS is a comprehensive system designed to:

Predict students’ literacy success and, for those found to be at risk:

To diagnose weaknesses, and

Set instructional objectives.

Because FAIR-FS is administered 3 times per year, it can be used to monitor growth in literacy skills for the purposes of revising instructional objectives.


7

Reliability & ValidityReliability & Validity

First 2 questions asked about measurement:– Does it actually measure what they say it measures?

– validity –

– Does it measure the same thing every time? – reliability –

Computer-adaptive format of FAIR-FS increases reliability (and efficiency) by tailoring the task to the student

Average reliability on FAIR-FS is .9 (very high)


8

ConclusionsConclusions

Different types of assessments are required to meet different purposes:– What decisions will be made from the

assessment data?

– Who will make those decisions?

Standardized tests are validated for specific purposes– Based on the types of analyses conducted


9

Development of the FAIR-FS

Development of the FAIR-FS


10

Components of Reading Comprehension

Components of Reading Comprehension

Word Recognition

Language Comprehension (vocabulary knowledge; syntactic knowledge)

World Knowledge

Motivation

Cognitive Abilities (e.g., working memory, executive functions)


11

Efficiency & ReliabilityEfficiency & Reliability

The FAIR-FS utilizes Computer Adaptive functionality to:– Maximize the amount of information provided

– Maximize reliability (and thus validity)

– Minimize testing time

Item Response Theory vs. Classical Test Theory– Reduces the number of items

– Increases the reliability


12

Fixed Item vs. CATFixed Item vs. CAT

Word Reading Grade 1

Low Ability Average Ability High Ability

1 see see see

2 me me me

3 run run run

4 look look look

5 you you you

6 did did did

7 that that that

8 was was was

9 got got got

10 back back back

Potentially Lower

Reliability High Reliability

Potentially Lower Reliability


13

Fixed Item vs. CATFixed Item vs. CAT

Grades K‐2


14

Grades K-2 FAIR-FSAdministration

Grades K-2 FAIR-FSAdministration


15

AlphabeticsPhonological Awareness (K)

Letter Sounds (K)Word ReadingSpelling (2)

Oral LanguageVocabulary Pairs

Following Directions

ComprehensionReading

ORListening Comp & Sentence

Comp

Probability of Literacy Success(PLS)

Computed

PLS<.85?

NO

YES

Grade Level

Diagnostic Tasks

STOP

STOP

Flow of TasksFlow of Tasks


16

What’s New?What’s New?

FAIR ‘09 FAIR‐FS

Format

1‐on‐1 with EST or AIRRequires paper test kit

1‐on‐1 with computer adaptive deliveryOffline AIR format also available

Tasks

Limited number of items & tasks Items targeted to student’s abilityMore oral language tasks

Comprehension

6 grade level passages per grade 26 passages available, spanning a wider difficulty range

Score Types

Provides probability of reading success Also measures growth in important skills

Diagnostics

17 tasks to target instruction 17 tasks administered on computer,eliminates need for paper test kit


17

Screening TasksScreening Tasks

Screening tasks are administered 1-on-1 and fall into 3 types:

1. Student and Teacher Task: Both the assessor and student see the screen, but only the teacher uses the mouse (e.g. LS)

or2. Student Task: Both see the screen, but the

student uses the mouse (e.g. FD)

3. Teacher Task: Only the assessor sees the screen and uses the mouse (e.g. PA)


18

Screening TasksScreening Tasks

All directions, practice items, & items delivered via audio

Each task computer-adaptive– Reduces administration time & frustration by only

administering items targeted to individual student’s ability level

– Total administration time for the screening will be approximately 15 minutes

Screening can be administered over several sessions if needed – If a task is started, ensure that task is completed

before ending the session


19

AlphabeticsSection 1

AlphabeticsSection 1

Kindergarten First Grade Second Grade

Fall Winter Spring Fall Winter Spring Fall Winter Spring

Letter Sounds

Phonological

Awareness

Word Reading

Spelling


20

Letter SoundsLetter Sounds

Time estimate 1 minute

Directions “You will see some letters on the screen. “Tell me the sound the letter makes.”

Practice item None

Prompt If student provides the long vowel sound, say to the student “That’s one sound that

letter can make, tell me a different one.” If the student produces the correct short

vowel sound, mark it as correct.

Report output Letter Sounds Ability Score and Percentile Rank

Letter Sounds & Phonological Awareness most predictive in Kindergarten

Student sees both uppercase and lowercase letters & pronounces sound (consonant, short vowel sound, common consonant digraph)

Each student will respond to between 5 and 29 sounds

Teacher marks correct/incorrect


21

Letter Sounds Letter Sounds


22

Phonological AwarenessPhonological Awareness

Time estimate 1 minute

Directions Listen as I say some words. If I say pig…tail, I know the word is pigtail.

Practice item “What would the word be if I say cup… cake?” If correct, say: “Yes,

the word is cupcake.”

Report output Phonemic Awareness Ability Score and Percentile Rank

Letter Sounds & Phonological Awareness most predictive in Kindergarten

Student hears an audio file pronounce a word that has been broken into parts/phonemes

Teacher marks correct/incorrect


23

Phonological AwarenessPhonological Awareness


24

Word ReadingWord Reading

Time estimate Less than 1 minute

Directions “Let’s see if you can read some words one at a time. Try to read each word and do the

best you can.”

Practice item None. If the student misses the first 7 words, the computer will automatically move to

the next task.

Prompt Encourage the student to take a guess if he/she is spending too much time on one word.

Report output Word Reading Ability Score and Percentile Rank

Single word decoding for grades 1 and 2; also available for Kindergarten students at AP3

One at a time, words appear on the screen for the student to pronounce; teacher indicates if the student correctly read the word (not timed)

Computer-adaptive format allows for a wide variety of difficulty for words


25

Word Reading Word Reading


26

Spelling(Grade 2 only)

Spelling(Grade 2 only)

Time estimate 3 minutes

Directions “I want you to spell some words. Listen carefully as each word and sentence are played aloud.

Some of the words will be easy and some may be hard. Do your best to spell each word

correctly.”

Practice item None; Test will discontinue if the first 7 words are misspelled.

Report output Spelling Ability Score and Percentile Rank; Student’s misspellings

Students will hear a word and will type to spell/sound‐out the word

To reduce frustration, this task will be computer adaptive, limiting the number of words that are too easy or too difficult

Students will be administered a minimum of five words and a maximum of 20 words

Score report will include student’s misspellings and a guide for analyzing errors will be available in the administration manual


27

SpellingSpelling


28

Oral LanguageSection 2

Oral LanguageSection 2



Vocabulary Pairs

Following

Directions


29

Vocabulary PairsVocabulary Pairs


Directions “Look at the boxes with words on the screen. Two of these go together. I

will name each one and you will click on the two that go together best.”

Practice item “Listen carefully as I name each one: book, toothbrush, toothpaste. Which

two go together best?”

Report output Vocabulary Pairs Ability Score and Percentile Rank

Requires students to match words that are semantically related (more reliable than expressive measures at this age)

Student hears words pronounced and clicks the words that go together

Some easier items have pictures


30

Vocabulary PairsVocabulary Pairs

Audio: “Listen carefully as I name each one: blue,

triangle, yellow. Which two go together best?”


31

Following DirectionsFollowing Directions


Directions “Look at the pictures on the screen. You will hear a sentence and I want you to click

the item named.”

Practice item Let’s try one. Click the book.

This time I want you to click the pictures in the order you hear them. Click the heart,

then click the plane.

For this item, you will have to move a picture. Let’s try one. Put the cat on the line.”

Report output Following Directions Ability Score and Percentile Rank

Student hears a sentence with directions (may be single or multi-step) and responds by selecting or moving objects on the screen

This type of attentional task has been found to be predictive of oral language abilities*


32

Following DirectionsFollowing Directions

Audio: “Click on the book, then click on the plane.”


33

ComprehensionSection 3

ComprehensionSection 3

This set of tasks is used to help develop a full student profile beyond measures in the screening to guide instruction aligned to LAFS.

Tasks include Listening, Reading, and Sentence Comprehension.



Listening

Comprehension

Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able

Reading

Comprehension

Sentence

Comprehension Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able

Avail‐

able


34

Listening Comprehension(Grade K only)

Listening Comprehension(Grade K only)

Screening task scores will be used to identify a passage of appropriate difficulty

Passages are informational and narrative

Five comprehension questions per passage (three explicit and two inferential)


Directions (Listening Comprehension) “Listen while I read __(title)__. When I’m finished, I will

ask you a few questions. Ready? Listen carefully.”

Practice item None

Report output Listening Comprehension Raw Score


35

Listening ComprehensionListening Comprehension


36

Reading ComprehensionReading Comprehension

Screening task scores (i.e., PLS) will be used to identify a passage the student is likely to decode

Passages are informational and narrative

The teacher will start a timer (on the computer) and mark miscues in order to capture fluency

Five comprehension questions per passage (three explicit and two inferential)


Directions (Reading Comprehension) “I would like you to read out loud for me. When you’re

done, I’ll ask you some questions about what you read. Read carefully and I will

use my stopwatch to tell how long it takes. The title of the story is__(title)___.”

Practice item None

Report output Reading Comprehension Raw Score

Number of words read correctly in 1 minute

Oral reading accuracy (as a percentage)


37

Reading Comprehension(Teacher Passage Screen)Reading Comprehension(Teacher Passage Screen)


38

Reading Comprehension(Teacher Passage Screen)Reading Comprehension(Teacher Passage Screen)


39

Sentence Comprehension(Grades K and 1 only)

Sentence Comprehension(Grades K and 1 only)


Directions “Look at the pictures on the screen. You will hear a sentence and I want you

to click the picture that best goes with the sentence.”

Practice item “Click on The baby is crying.”

Report output Sentence Comprehension Ability Score and Percentile Rank

Student hears a sentence and clicks the one picture (out of four) that best fits the sentence

Syntactic abilities and listening comprehension at the sentence level is found to be an important predictor of reading comprehension*


40

Sentence ComprehensionSentence Comprehension

Audio: “Click on: The bird is flying toward the nest.”


41

AlphabeticsPhonological Awareness (K)

Letter Sounds (K)Word ReadingSpelling (2)

Oral LanguageVocabulary Pairs

Following Directions

ComprehensionReading

ORListening Comp & Sentence

Comp

Probability of Literacy Success(PLS)

Computed

PLS<.85?

NO

YES

Grade Level

Diagnostic Tasks

STOP

STOP



42

Grade Level Diagnostic TasksGrade Level Diagnostic Tasks

Computer administers these tasks to students whose PLS on the screening tasks was below .85

Used to assist teachers in selecting skills to target for instruction

Number of subtests administered depends on student’s skill

If the student demonstrates mastery of a skill, that subtest is not administered at subsequent APs

Computer-administered, but not computer-adaptive


43

Task Kindergarten Grade 1 Grade 2

Print Awareness

Letter Name Knowledge

Letter Sound Knowledge (AP3)

Phonological Blending

Letter Sound Connection – Initial Sounds

Letter Sound Connection – Final Sounds

Phonological Deletion – Parts and Sounds

Phonological Deletion – Initial

Phonological Deletion – Final

Word Building – Initial Consonants

Word Building – Final Consonants

Word Building – Consonants

Word Building – Vowels

Word Building – CVC/CVCe

Word Building – Blends

Word Building – Blends & Vowels

Multisyllabic Word Reading


44

FAIR-FS Grade 3 – 12Administration

FAIR-FS Grade 3 – 12Administration


45


No

Yes

WrittenResponse

WordRecognition(about2min.)

ComputeProbabilityofLiteracy

Success(PLS)

SyntacticKnowledge(about5min.)

VocabularyKnowledge(about3min.)

ReadingComprehension(about15min.)

STOP

IsPLS<.85?

STOP

Takeoptionaltasks?

No

Yes

OralReadingFluency

OralResponse

Paper/pencil administration

Computer administration


46

What’s New in 3 – 12?What’s New in 3 – 12?

FAIR Classic FAIR‐FS

Tasks

Reading ComprehensionMazeWord Analysis

Word Recognition (WR)Vocabulary Knowledge (VK)*Syntactic Knowledge (SK)*Reading Comprehension (RC)

RC Passage Placement

Starts with grade level passage Ability in WR & VK determines initial passage

Probability of Success

Calculated based on RC & prior FCAT Based on current RC, WR, & VK scores;Predicts to SAT‐10 reading comprehension

Additional tools (optional)

Ongoing progress monitoringDiscussion templates

Open Response DiagnosticsOngoing progress monitoring still available


47

Word Recognition TaskWord Recognition TaskScreen#1

WordRecognitionTask

The student hears a word pronounced by the computer.

The student selects the word pronounced by the computer.


48

Vocabulary KnowledgeVocabulary KnowledgeScreen#2VocabularyKnowledge

The student reads the sentence on the screen.

The student completes the sentence with 1 of 3 morphologically related words.


49

Reading ComprehensionReading ComprehensionScreen#3Reading

Comprehension

The student reads the passage, then clicks to show the questions. Questions & passage can be viewed simultaneously.

The student selects the correct response to the question.


50

Diagnostic: Syntactic Knowledge (Students with PLS <.85 will complete this task, optional if >.85 )Diagnostic: Syntactic Knowledge (Students with PLS <.85 will complete this task, optional if >.85 )

The student hears the sentence read by the computer.

The student selects the word that best completes the sentence.


51

Optional Diagnostic TasksOptional Diagnostic Tasks

Teacher has the option to administer these tasks to students after they have completed the Syntactic Knowledge Task

Open response items allow teacher to analyze individual students’ approach to answering questions

Tasks are mostly teacher-administered and teacher-scored

These scores are not entered in the PMRN


52

Optional Diagnostic: Open ResponseOptional Diagnostic: Open Response

14 – 16 passages for each grade

Some Literary, some Informational

Text complexity (quantitative & qualitative) fit the grade bands recommended by Nelson, Liben, Liben, & Perfetti

The teacher will choose the passage with associated questions to administer (print from online system)

Each passage has three (3) oral response comprehension questions and one (1) written response comprehension question

WrittenResponse

OralReadingFluency

OralResponse


53

Optional Diagnostic:Oral Reading Fluency (ORF)

Optional Diagnostic:Oral Reading Fluency (ORF)

Directly aligned to Reading Foundation strand

Student reads passage aloud while the teacher – Marks miscues and

– Scores Oral Reading Fluency• Rate (total words read correctly in 1 minute)

• Accuracy (WRC/total words read)

• Expression (rating on the NAEP rubric)


54

Administering and scoring oral response comprehension questionsAdministering and scoring oral response comprehension questions

Directly aligned to RI, RL, and L strands of the LAFS

The teacher reads each question to the student (while the student follows along)

There is space for the teacher to record the student’s oral response

A 4-point rubric and sample answers for each category are provided

OralResponse


55

Administering Written Response Comprehension QuestionAdministering Written Response Comprehension Question

The student will respond to one (1) written question related to the passage s/he just read (typed response)

Question will target one of the following types of writing:– Opinion/argumentative

– Informative/explanatory

– Narrative

Student will log back into the web application & instructions will be provided

Make sure the student also has:– Hard copy of the passage

– Scrap paper & pencil for planning purposes

– Headphones

WrittenResponse


56

Scoring the Written Response Comprehension Question

Scoring the Written Response Comprehension Question

Teachers will be able to print out 2 documents from the written response to score:– The sample at 5 minutes for scoring writing fluency

• Total number of words written (TWW) will be counted and provided by the computer

• Number of correct writing sequences minus incorrect writing sequences (CIWS) needs to be hand scored

– The complete sample that is collected at 10 minutes • Scored utilizing the FAIR-FS checklist for the written response

WrittenResponse


57

Scoring and ReportsScoring and Reports


58

Scoring and ReportsScoring and Reports

Important note:

Reports from FAIR-FS were designed to facilitate instructional decision making including, systems level problem-solving and data-based decision

making.

FAIR-FS scores are not intended to be the sole data point in determining

retention or special education determination


59

Score Reports will be available online

Score Reports will be available online

Detailed reports for teachers and parents– Includes profile of student scores

Graphs to show ability scores in relation to average scores for that grade level

Computer adaptive tasks:– Ability scores

– Percentile ranks

Probability of Literacy Success


60


61


62

Probability of Literacy Success (PLS)

Probability of Literacy Success (PLS)

Score represents the likelihood that a student will score at the 40th percentile on the end-of-year outcome measure (i.e., SAT-10, AIR)

Grades 3-12: PLS is based on aggregate of WRT, VKT, and RCT

Grades K-2: PLS is based on an aggregate of scores on Alphabetics and Oral Language tasks

PLS of .50 predicts that student has 50/50 chance of achieving the passing

score on the outcome measure


63

Ability ScoresAbility Scores

Scores represent an estimate of ability in a specific skill and reflects true change over time as ability increases or decreases– Covers a range of ability from 3rd grade to 10th

– Scores range from 150 – 1000 (mean of 500 and standard deviation of 100)

Indicates degree of growth for each student

A 3rd grade student with an ability score of 500 is

performing exactly the same as a 5th grader with an ability score

of 500


64

Percentile RanksPercentile Ranks

Score is used to rank one student’s performance in relation to a particular group of other students– Ranges from 1 – 99 (25th through 75th percentile

represents the expected scoring range)

– Based on a representative sample of Florida students

2nd grade student with a percentile rank of 55 performed

better than 55% of other 2nd graders in Florida


65

Considerations for GrowthConsiderations for Growth

Ability scores are on an equal interval scale whereas percentile rank is not

Percentile rank is relative to other student’s performance & PLS is relative to another assessment. Ability score does not involve a comparison


66

Score Types for Computer-Adaptive Tasks

Score Types for Computer-Adaptive Tasks

Score type What it reflects What it does NOT reflect

Ability score • Quantifies a student’s level of skill and reflects changes

• Scale ranges from a minimal amount of skill to expert

• Performance compared to other students

• Grade‐level performance

Percentile rank • Student’s ability comparedto other students in the same grade

• Percentage of correct responses

• Growth• Level of expected

performance

Probability of Literacy Success

• Likelihood the student willreceive a passing score on end‐of‐year test

• Growth• Previous year’s end‐of‐

year test score


67

Score ProfileScore Profile

Generally, a skill should be targeted for instruction when scoring below the 30th percentile

The lower bars represent skills that are relative weaknesses for a student and higher bars indicate relative strengths

WR= Word RecognitionVK = Vocabulary KnowledgeRC = Reading ComprehensionSK = Syntactic Knowledge

5

15

25

35

45

55

65

75

85

95

WR VK RC SK

Percentile Ran

k

Computer‐adaptive Tasks


68

Student Score ReportsStudent Score Reports


69

Scores by Grade LevelScores by Grade Level

Developmental A

bility Score

Developmental A

bility Score


70

Communicating with ParentsCommunicating with Parents

Computer-generated parent resource letters will be available

Letters will contain information on strength and weaknesses, progress over the school year, and skills targeted for instruction

Letters will also include resources on strengthening reading skills assessed in FAIR-FS


71

Instructional Implications Instructional Implications


72

Informing Instruction Informing Instruction

Use the student’s score profile and the classroom report to identify students and skills for supplemental instruction/intervention– Identify lower performing students from the classroom

report

– Determine instructional needs based on profiles of strengths and weaknesses:

• Supplemental curriculum

• Added emphasis on particular skills in small group*

• Target skills that are relative weaknesses during center time**


73

Teaching Beginning ReadingTeaching Beginning Reading

Teach PA, LN, & LS, and practice blending to read simple words; “build” simple words

Provide sequential, explicit instruction in letter-sound & sound-spelling patterns; teach high frequency regular & irregular words. Practice in isolation & in text

Teach word analysis with syllable patterns and multiple syllables; practice

Provide daily text reading with & without feedback with attention to accuracy, fluency, & comprehension


74

Teach Academic Language SkillsTeach Academic Language Skills

Have discussions that entail inferential language, including read-alouds

Explicitly teach narrative language skills (e.g., how to summarize by retelling goal, conflict, resolution or how to make logical predictions, modeling linguistic elements or narrative structure)

Teach academic language in the context of reading and/or listening activities


75

Case StudyCase Study


76

Ms. Dunphy’s 2nd period classMs. Dunphy’s 2nd period class

Ms. Dunphy (a grade 4 teacher) takes her class to the computer lab to take the FAIR-FS during AP1

They log in to the PMRN

And respond to 3 tasks:– Word Recognition Task (~ 2 minutes)

– Vocabulary Knowledge Task (~ 3 minutes)

– Reading Comprehension (1-3 passages with questions)


77

Ms. Dunphy’s 2nd period classMs. Dunphy’s 2nd period class

25 students took FAIR‐FS screening

5 students identified as “at‐risk” and take Syntax task

Ms. Dunphy administers Open Response tasks to 3

students


78

Oral Reading Fluency– Reading Foundational

Skills Strand

Oral Response– Reading for Information

Strand; Reading Literary Text Strand; Language Strand

Written Response– Writing Strand;

Language Strand

WrittenResponse

Takeoptionaltasks?

Yes

OralReadingFluency

OralResponse

Ms. D. Administers Open Response TasksMs. D. Administers Open Response Tasks


79


1. Stanley reads story aloud while Ms. Dunphy scores for accuracy, rate, & expression

Marks end of 1

minute

Marks end of 1

minute

Marks MiscuesMarks Miscues

Calculates accuracy and rate

Uses a rubric to rate expression

OralReadingFluency


80


2. Ms. Dunphy asks Stanley 3 questions about the story and records Stanley’s oral response. (Stanley also has a copy of the passage and questions).

OralResponse


81


3. Ms. Dunphy makes sure Stanley has his packet, a pencil, & headphones and directs him to log in to the PMRN

This task will take approximately 20 minutes.


82

Scoring Stanley’s Written Response

Scoring Stanley’s Written Response

Obtain Stanley’s written response from the PMRN– Writing at 5 minutes

– Completed writing (10 minutes)

Score the 5-minute sample using Appendix B of the administration manual

Score the 10-minute sample using the holistic writing checklist


83

QuestionsQuestions


84

• Curriculum questions: contact your district reading office

• Technical questions: • See the FAIR-FS/PMRN Course (Coming Soon!)

• Call or email the PMRN help desk http://www.fcrr.org/pmrn/pmrn.asp

• Content questions: contact Just Read, Florida! at 850-245-0503 http://www.fldoe.org/

For More Information

Documents

FAIR FS Preview webinar 5/16/2014 - Florida State University · FAIR‐FS Preview webinar 5/16/2014 4 4 Teacher: Are my students making progress? How do I differentiate instruction?