AZELLA Update Marlene Johnston and Linda Harvey OELAS Conference December 7, 2011

AZELLA Update

Marlene Johnston and Linda HarveyOELAS Conference

December 7, 2011

Agenda for Today

• Update on AZELLA Development Process

• Statewide ELL data

AZELLA RevisionUpdate

Framework for High-Quality English Language Proficiency Standards and Assessment

Building on the best knowledge from relevant research and practice, the Framework provides criteria for high-quality ELP standards and aligned assessments.

Arizona Department of Education, Office of English Language Acquisition Services

Test Characteristics

• Criterion referenced test of ELP Standards• Grades K-12 tested in 5 Stages:

– K, 1-2, 3-5, 6-8, 9-12• 5 Proficiency Levels:

– Pre-emergent, Emergent, Basic, Intermediate, Proficient

• 4 Domains tested:– Listening, Speaking, Reading, Writing

6

ELL Assessment History

School Years 2005-2006Statewide SELP implementation

School Years 2007-2009AZELLA Form AZ-1

School Year 2009-2012AZELLA Form AZ-2


Governing Statutes and Authority

Federal Title I Requirements for Assessment(20 U.S.C. Sec. 1111). (3) Academic Assessments.-State Plans(7) ACADEMIC ASSESSMENTS OF ENGLISH LANGUAGE PROFICIENCY

Provide for an annual assessment of English proficiency measuring the oral language, reading and writing skills in English of all ELL students.

Federal Title III Requirements for Assessment20 USC 6841 SEC. 3121 Evaluations(d) EVALUATION MEASURES

A state shall approve evaluation measures that are designed to assess the progress of children in attaining English proficiency, including a child’s level of comprehension, speaking, listening, reading, and writing skills in English.

Governing Statutes and Authority

Arizona Revised StatutesARS 15-756 (B)Identification of ELLs

The English language proficiency of all pupils with a primary or home language other than English shall be assessed through the administration of English language proficiency assessments in a manner prescribed by the superintendent of public instruction. The test scores adopted by the superintendent as indicating English language proficiency shall be based on the test publishers' designated scores. The department shall annually request an appropriation to pay for the purchase of all language proficiency assessments, scoring and ancillary materials as prescribed by the department for school districts and charter schools.

Adherence to Standards

AZELLA development follows Standards for Educational and

Psychological Testing (AERA, APA, NCME, 1999)

American Educational Research Association, American Psychological Association,

National Council on Measurement in Education

Stakeholder Roles and Responsibilities

ADE • Develop a valid and reliable test• Provide needed materials and training for

administration• Assure accurate test scoring and timely delivery

of test resultsLEAs• Timely and accurate administration of tests to

PHLOTE, ELL and FEP students

Who are Arizona’s ELL

students?

12

Demographics - Arizona

Approximately 10% of Arizona’s K-12 students are English Language Learners (ELL).

•116,506 ELL Students in 2010•76,320 ELL Students in 2011 (preliminary)


13

Percent of ELLsBy Grade Span

• K-2

• 3-5

• 6-8

• 9-12

• 47%

• 25%

• 15%

• 13%Arizona Department of Education, Office of English Language Acquisition Services

14

“Top Ten” Districts

10 Districts

42% of all ELLs statewide


15

Top 5 Languages in Arizona other than English

Spanish 80%

Navajo 2%

Vietnamese >1%

Arabic >1%

Somali >1%


16

Arizona ELL Data

about 70% of Arizona ELL students

are at Intermediate achievement level


Purpose of AZELLA

The purpose of AZELLA is to measure language proficiency in order to:

• determine student placement for ELL services.• measure student progress in English language

development.• establish exit criteria for students.• monitor the language proficiency of FEP

students.

Uses of AZELLA Data for Accountability

• Provides data for AMAO determinations at the district and state levels.– AMAO 1 - Making progress – AMAO 2 – Reclassification

• Contribute to Schools Labels A-F

• Longitudinal data analysis of ELL performance

Implications of AZELLA results

• Student placement for services• Student exit from program participation• District and school performance measures for

accountability purposes (federal and state)• Supplemental funding for ELL services• Indications of program effectiveness

Proficient 93,356 46%

Intermediate 62,397 31%

Basic 31,866 16%

Emergent 5,906 3%

Pre-Emergent 8,355 4%

Counts of ALL Tests AdministeredFY 2010 (Including IFEP/FEP, multiple administrations)

Source: ADE, Data Mart ELL 72 Table. All valid assessments, including multiple attempts by individual students are included. Data was drawn on 1/26/2011.

What is the Content?

• English Language Proficiency Standardshttp://www.azed.gov/english-language-learners/2011/07/20/finalized-english-language-proficiency-elp-standards/

• Link to Academic Standards and CCSShttp://www.azed.gov/standards-practices/common-core-state-standards/

http://www.azed.gov/english-language-learners/2011/07/20/finalized-english-language-proficiency-elp-standards/



http://www.azed.gov/standards-practices/common-core-state-standards/

http://www.azed.gov/standards-practices/common-core-state-standards/

What is English Language Proficiency ? (ELP)

English language proficient students demonstrate a “good working knowledge of English and are able to do regular school work in English.”

A.R.S. 15-752

English language proficiency as a single construct

• Arizona defines English Language Proficiency as a single construct.

• Listening, Speaking, Reading and Writing are the four modalities that contribute to this construct.

• The relative contribution of each of the

modalities changes based on proficiency and grade. AZELLA blueprint reflects these changes.

Draft AZELLA Blueprint

• Stage I – Reading and Writing represent 53%

• Stage II – Reading and Writing represent 57%

• Stage III – Reading and Writing represent 57%

• Stage IV – Reading and Writing represent 60%

• Stage V - Reading and Writing represent 61%

Enhanced Assessment Grant

• The purpose is to examine the relative contributions of the modalities to one another and to English language proficiency.

• Reporting includes a statistical technique called confirmatory factor analysis to examine the contributions of the modalities.

• Reporting is based on 2009-2010 data using modality raw scores and AIMS scores where available.

Listen Speak Read

0.84

ELP

0.88

0.16 0.29 0.23

0.91

Write

0.94

0.11

Construct of ELP ELP is a good predictor of scores in sub-domains

This is one model for ELP that appears to fit AZ-2 data well and explains student performance on the modality test. Can be applied to all grades 2-12. Chart reflects Grade 9 examinees.

AZELLA Oral

ListenSpeak Read

0.210.82

AZELLA Comprehension

0.85

0.48

0.330.52 0.28

AIMS Language Arts

Read Write

0.86 0.68

0.540.27

0.92

0.48

0.57

Relationship between AZELLA Oral/ Comprehension/AIMS for Grade 5

There is a positive relationship between AZELLA AZ-2 comprehension and AIMS Language Arts

Goals of AZELLA Revision• Aligned to the new ELP Performance Indicators • New Blueprint reflects slight modification to weighting based on

ELP Standards – Kindergarten emphasis is Listening & Speaking– H.S. emphasis is Reading & Writing

• Own high performing items for 2 test versions• Kindergarten

– Placement Test– Exit test stronger alignment to 1st grade content

• FEP test – more efficient/based on Intermediate skills• Annual test in Spring testing window• Structured to allow for more uniformed administration• Publish sample items

Test Development

2011• Advisory Group Meetings• AZELLA test blueprints developed• Item specifications for AZELLA• Item Writing (over 1300 items)• Content & Bias Review• ADE/WestEd content experts review of itemsfor content and universal access considerations

Types of Items

• Reading – Multiple choice and short answer open-ended oral responses

• Writing – Multiple choice and extended open-ended responses

• Listening – Multiple choice responses• Speaking – Open-ended short and extended

responses

Field Test (What… Who… When)

What• A field test of new AZELLA test items is needed to ensure they

meet the expectations necessary for inclusion on the new AZELLA test.

• Results of the field test are not used to determine student English proficiency levels.

• No results of the field test provided to districts, charters, schools, parents, or teachers.

• Stages II-IV tests Fall 2011; Stage I tests Spring 2012

Field Test

Who • 310 districts; 1165 schools• Approximately 32,000 tests administered• ELLs identified in the sample plus a select

sample of non-ELLs (never ELL + FEP)

When• November 7, 2011 through December 2, 2011

Test Administration: General AZELLA Field Test Format

Session Domain Type of Administration Special Equipment for Administration

Session 1

Listening: Multiple Choice

Individually or group administered with paper and pencil

Audio CD and CD player

Break

Session 2

Writing: Writing Production


None

Break

TADP. 3

Test Administration: General AZELLA Field Test Format (continued)

Session Domain Type of Administration Special Equipment for Administration

Session 3

Reading: Multiple Choice


None

Break

Session 4

Speaking: Oral Response Individually administered

Speaker telephone

and computer

TADP. 4

Speaking Test Administration

• Collaborative development• Field test responses captured by telephone• Open ended items• Repeat items• Scored by trained human scorers based on

rubrics• Comparability study to determine reliability of

computer scoring

Upcoming/Scoring

• Scoring – Quality controlWriting

• Conventions • Open ended responses

– Scored at Pearson Scoring Center in San Antonio– Scorers trained during 3 week period beginning January 2012

» Use of rubrics

Speaking• Open ended responses

– Scored at Pearson Scoring Center in San Antonio– Scorers trained during 3 week period beginning January 2012

» Use of rubrics

Upcoming Activities

• Data Analysis – March 15-16, 2012

• Kindergarten field test– Spring 2012

• Standard Setting– March 2013

• Alignment Study– Spring 2013

Proposed Reporting Categories

• Summary – State/District/School• Student Level

• Composite• Listening • Speaking • Reading• Writing• Literacy score (reading and writing)• Comprehension (reading and listening)• Oral (speaking and listening)• Language Strand

Validity and Reliability

Measures of Validity• Internal structure—“the degree to which the relationships

among test items and test components conform to the construct on which the proposed test score interpretations are made.” (AERA, APA, NCME, 1999, p. 13) Each task in the assessment should contribute positively to the total result.

• Reliability is the degree to which scores remain consistent over an assessment procedure (Nitko, 2004). Data analysis and psychometric components will include measures that are essential to proper evaluation and use of the assessment.

Validity and Reliability (continued)

Other Elements of Validity• Content—test represents an appropriate sampling of skills,

knowledge, and understanding of the domain tested. Practical considerations for test length, administration time and effort, scoring time and cost

• Educator Input—ELL teacher committees promotes face validity. Content and bias review processes and item level data review contribute to validity.

• Universal design principles promote the inclusion of the widest ranges of student and support the validity of inferences about proficiency.

Validity and Reliability (continued)

Other Elements of Validity• Materials development and production

components utilize rigorous processes.• Strict attention to the development and refinement

of Directions for Administration are crucial to support validity.

• Open-Ended scoring utilizes detailed rubrics and rigorous training for scorers. Inter-rater reliability is controlled by read behinds and other quality control measures.

ResearchLanguage Test Construction and EvaluationJ. Charles Alderson, Caroline Clapham, Dianne Wall, Cambridge University Press

Language Testing in PracticeLyle F. Bachman, Adrian S. Palmer, Oxford University Press

The Language Demands of SchoolEdited by Alison L. Bailey, University of California, Los Angeles

Comment: Making an Argument for Design Validity Before Interpretive ValidityDerek C. Briggs, University of Colorado

Assessing ListeningGary Buck, University of Michigan, Lidget Green, Inc.

Designing and Analyzing Language TestsNathan T. Carr, California State University

ResearchDoes an Argument-Based Approach to Validity Make a Difference?Carol A. Chapelle, Iowa State University, Mary K. Enright, Educational Testing Service,

Joan Jamieson, Northern Arizona University

Threats to the Valid Use of AssessmentsTerry J. Crooks, University of Otago, Michael T. Kane, University of Wisconsin-Madison,

Allan S. Cohen, University of Wisconsin-Madison

Handbook of Test DevelopmentEdited by Steven M. Downing, University of Illinois, Thomas M. Haladyna, Arizona

State University

An Argument-based Approach to ValidationMichael T. Kane, ACT

ResearchValidating High-Stakes Testing ProgramsMichael Kane, National Conference of Bar Examiners

Does an Argument-Based Approach to Validity Make a Difference?Carol A. Chapelle, Iowa State University, Mary K. Enright, Educational Testing Service, Joan

Jamieson, Northern Arizona University

Test Validity: A Matter of ConsequenceSamuel Messick

Consequences of Test Score Use as Validity Evidence: Roles and ResponsibilitiesPaul D. Nichols and Natasha Williams, Pearson

Evaluation of Validity and Validation: Quality criteria for the evaluation of validity and validation of tests.

Sakia Wools, CITO

AZELLA Testing TimelineDate(s) Test Purpose Test Instrument

November 7 to December 2, 2011 AZELLA Item Field Test – in selected schools

Fall 2011 AZELLA Field Test

February and March 2012 Monitored FEP1 and FEP2 students annual assessment

AZELLA Form AZ-2

March and April 2012 Continuing ELL students Stages I - V annual assessmentNote: tests must be submitted for scoring by 04-27-12

AZELLA Form AZ-2

April 23 to May 11, 2012 AZELLA Field Test of Kindergarten Stage I – in selected schools

Spring 2012 AZELLA Field Test

School Year 2012-2013 New student placements for Stage I AZELLA Kindergarten Screener

School Year 2012-2013 New student placements for Stages II - V

AZELLA Form AZ-2

January and February 2013 Testing of monitored FEP1 and FEP2 students

AZELLA Form AZ-2

January and February 2013 Testing of continuing ELL students Stages I - V

Revised AZELLA

Comments regarding

the field test

47

Statewide Evidence of

Success

Statewide Reclassification Rate

2005-2006 15%

2006-2007 12%

2007-2008 22%

2008-2009 29%

2009-2010 30%

2010-2011 33.5%

49

Description of SEI Students for this Study

• Students must have participated only in the SEI/ILLP program within the given year.

• Students must have a qualifying assessment.

• Students who are excluded from the research: withdrawn by parent request or by a specific special education determination.

• Cohort is defined as students who enter the SEI/ILLP program for the first time.

• Fiscal Year (FY) is defined as school year (e.g., FY08 is school year 2007-2008).

50

STATEWIDE ELL STUDENT PERFORMANCE

BY COHORT YEAR

51

Year Total Pre-Emergent Emergent Basic Intermediate Proficient

Cohort08 26313 741 660 3757 10601 10554

FY2009 *15018 45 86 1898 8622 4367

FY2010 10470 1 7 453 4005 6004

FY2011 4354 2 3 305 2875 1169

Total Proficient FY08-FY11 22094

Cohort 2008Defined as K-12 ELL students who entered the SEI/ILLP program in the 2007-2008 school year and stayed in the program until they either graduated, withdrew, or were classified proficient.

*The FY totals do not include students that graduated or tested proficient from the previous year.

FY2008 FY2009 FY2010 FY20110

5000

10000

15000

20000

25000

10554

14921

20925 22094

Number of Students Tested Proficient*Cohort 2008 – 26,313 students

Proficient(84% across years)


*These are cumulative values across years


Cohort09 24109 347 351 2118 8591 12702

FY2010 *10984 56 95 2261 5325 3247

FY2011 7636 10 18 610 3430 3568

Total Proficient FY09-FY11 19,517



FY2009 FY2010 FY20110

5000

10000

15000

20000

25000

0

12702

1594919517

Number of Students Tested Proficient*

Cohort 2009 – 24,109 students

Proficient(81% across years)




Cohort10 19828 678 581 5010 8479 5080

FY2011 *14541 316 259 2833 6962 4171

Total Proficient FY10-FY11 9251



FY2010 FY20110

5000

10000

15000

20000

25000

0

05080

9251


Cohort 2010 – 19,828 students

Proficient47% across years



STATEWIDE ELL STUDENT PERFORMANCE

BY COHORT YEAR AND GRADE

FY2008 FY2009 FY2010 FY20110

5000

10000

15000

20000

25000

828310923

15875 16727


Cohort 2008: K-2 – 19,429 students


Cohort 2008: K-2Defined as ELL students - Grades K-2 who entered the SEI/ILLP program in the 2007-2008 school year and stayed in the program until they either withdrew or were classified proficient.


FY2008 FY2009 FY2010 FY20110

500

1000

1500

2000

2500

3000

868

1692

22182425

Number of Students Tested Proficient*Cohort 2008: 3-5 -- 2599 students


Cohort 2008: 3-5Defined as ELL students - Grades 3-5 who entered the SEI/ILLP program in the 2007-2008 school year and stayed in the program until they either withdrew or were classified proficient.


FY2008 FY2009 FY2010 FY20110

500

1000

1500

2000

2500

3000

793

13271690 1768


Cohort 2008: 6-8 -- 1959 students


Cohort 2008: 6-8Defined as ELL students - Grades 6-8 who entered the SEI/ILLP program in the 2007-2008 school year and stayed in the program until they either withdrew or were classified proficient.


FY2008 FY2009 FY2010 FY20110

500

1000

1500

2000

2500

3000

610979

1142 1174


Cohort 2008: HS -- 2326 students


Cohort 2008: HSDefined as ELL students - Grades HS who entered the SEI/ILLP program in the 2007-2008 school year and stayed in the program until they either withdrew or were classified proficient.


62

Source: Haas, E. (July 2010) Assessing Achievement of English Language Learners: Pass-Fail Status on Arizona’s Language and Content Tests REL West Analysis, WestEd

AZELLA / AIMS Correlation

63

Percent Passing AIMS 2010

Non-ELL (neither FEP nor ELL)

FEP All Years ELL

Subject Not Low-SES Low SES Not Low-SES Low SES Not Low-SES Low SES

Reading 85% 68% 79% 76% 24% 23%

Writing 82% 65% 73% 71% 25% 27%

Math 71% 49% 58% 56% 19% 19%

Large Achievement Gap Low Achievement Gap Low Achievement Gap


Questions?

[email protected]@azed.gov

http://www.azed.gov/english-language-learners/

mailto:[email protected]

mailto:[email protected]

Documents

AZELLA Update Marlene Johnston and Linda Harvey OELAS Conference December 7, 2011