Embed Size (px)
Citation preview
WE GAVE THE TEST -NOW WHAT?
ANALYSIS AND REPORTING FOR THE SOUTH CAROLINA ARTS ASSESSMENT PROGRAM
Ashlee A. Lewis, Office of Program EvaluationR. Scot Hockman, SC Department of Education
Intent(s): To demonstrate the process by which the SC Arts Assessment Program moves
from test administration to data analysis to reporting results to schools.
To talk through the continuous improvement process used for the SCAAP.
SOUTH CAROLINA ARTS ASSESSMENT PROGRAM
(SCAAP)
https://scaap.ed.sc.edu
SCAAP Collaborators
South Carolina Department of Education (SCDE) Funding Agency
Office of Program Evaluation (OPE) at USC Test And Measurement Specialists Logistics
South Carolina Arts Educators Content-area Experts Capacity building through statewide arts
assessment institutes
Current SCAAP Assessments
Six different assessments in various stages: Four Entry Level Assessments
Music & Visual Arts (pilot tested in 2002) Dance & Theatre (pilot tested in 2005)
Two Middle Level Assessments Music & Visual Arts (pilot tested in 2008)
Current SCAAP Assessments
All assessments… are aligned to SC Academic Standards for
Visual & Performing Arts (2010) have two sections
Multiple-choice/Selected Response Section (45 items)
Performance Task Section (2 tasks)
SCAAP Administration
Administered each spring to schools that received Distinguished Arts
Program (DAP) grants from the SC Department of Education
Administered at individual schools by appointed test administrators trained by the
SCAAP team
Test administrators trained online and in person by the SCAAP team
SCAAP Participants
2004—66 schools with approximately 5,200 students 2005—51 schools with approximately 3,700 students 2006—70 schools with approximately 4,900 students 2007—81 schools with approximately 5,800 students 2008—56 schools with approximately 4,400 students 2009 – 46 schools with approximately 3,500 students 2010 – 41 schools with approximately 3,740 students 2011 - 45 schools with approximately 3,540 students 2012 – 32 schools with approximately 2,545 students 2013 – 34 schools with approximately 2,763 students 2014 - 34 schools with approximately 2,572 2015 - 41 schools with approximately 3,700 students
(Expected)
Music Test Specifications
StandardOverall
Emphasis
Percent covered by Assessment
Format
Selected
Response
Performance
Tasks
Standard 1: Performance 25% - 100%
Standard 2: Creating Music 20% 25% 75%
Standard 3: Music Literacy 25% 100% -
Standard 4: Critical Response
to Music
15% 100% -
Standard 5: History and
Culture
10% 100% -
Standard 6: Connections 5% 100% -
Visual Arts Test Specifications
Standard Overall
Emphasis
Percent covered by
Assessment Format
Selected
Response
Performance
Tasks
Standard 1: Creating Art 25% 40% 60%
Standard 2: Structures and Functions 25% 50% 50%
Standard 3: Exploring Content 10% 100% -
Standard 4: History and Culture 10% 100% -
Standard 5: Interpreting Works of
Visual Art
25% 25% 75%
Standard 6: Connections 5% 100% -
MULTIPLE CHOICE/SELECTED RESPONSE
https://scaap.ed.sc.edu
The Assessment Process – Achieving the Arts Assessment Mission
(Brophy)
Establish Goals, and Outcomes
Develop and implement
assessments
Collect Assessment
Data
Interpret and Evaluate the
Data
Modify and Improve
Impr
ove
teac
hing
and
le
arni
ng
Assessment
The Assessment Process – Achieving the Arts Assessment Mission
(Brophy)
Establish Goals, and Outcomes
Develop and implement
assessments
Collect Assessment
Data
Interpret and Evaluate
the Data
Modify and Improve
Impr
ove
teac
hing
and
le
arni
ng
AssessmentArt
s Educa
tion
Mis
sion
SCAAP Item Bank
Example Music Item
Example Visual Arts Item
Analyses Performed
Reliability indices for test forms Cronbach’s alpha and corrected split-half index
Test form equating Using Item Response Theory (IRT) Cross year and cross form test equating Empirical reliability based on fitted IRT model
Differential item functioning (DIF) analysis for gender and ethnicity
Distribution of p-values (percent correct) for items
Discrimination indices for each item
Reliability Estimates
Test Empirical
Reliability
Form # of Item
s
Cronbach’s
Alpha
Corrected Split
Half
Music .831 45 0.79 0.79
2 45 0.82 0.82
Visual Arts
.861 45 0.85 0.86
2 45 0.86 0.85
Item Review Process
Convene arts advisors in fall to revise items identified as problematic.
Review and revise based on: P-values Discrimination indices Differential Item Functioning (DIF) Distribution of distractors
Archive items Write new items to incorporate into test
forms
Improving the Assessment
Each year, item analysis guides the revisions of the assessment for the following year: P-values: Between .20 and .85 Discrimination index: .19 or higher Differential Item Functioning (DIF): All items
which receive a C classification are examined.
PERFORMANCE TASKS
https://scaap.ed.sc.edu
Performance Tasks
Visual Arts Compare and contrast two artworks using art
terms Create a drawing with given art
elements/principles based on a prompt Music
Sing a familiar song Improvise an 8-beat rhythm pattern using
rhythm syllables and maintaining a steady beat
Preparing Performance Tasks
Receive tasks from schools (flash drives and booklets)
Scanning visual arts performance tasks Mass uploading of tasks to website for online
rating
Performance Task Benchmarking
A sampling of responses is chosen and “pre-scored” by members of the SCAAP team based on the rubrics
Arts advisors indicate their agreement or dissent with existing scores
Discrepancies in scores are discussed and elements of the rubric are clarified
Validated, benchmarked tasks are used to train and monitor raters
Rater Training
Rater trainings have been done online and in person, depending on level of experience
Raters review rubrics and anchor items and review rater bias types
Interactive practice tests provide feedback on scoring
Raters must pass qualifying test before they begin rating
Rating System
Entire rating system is online Raters must pass a refresher test after
scoring 100 tasks Monitoring: Seeds are placed randomly
throughout un-scored tasks On average, each rater scores 600 tasks
over the course of 3 weeks
Analyses performed
Inter-rater reliability (GENOVA) Both MC and Performance tasks:
Correlations between multiple-choice test forms and performance tasks for each area.
Inter-rater reliability estimatesMusic 2013
Performance Task
CriteriaGeneralizab
ilityCoefficient
Index of Dependabi
lity
1 (Singing)
Tonal 0.94 0.94
Rhythm 0.84 0.83
Vocal 0.87 0.87
2 (Improvisati
on)
Rhythm 0.87 0.87
Improvisation
0.79 0.79
Inter-rater reliability estimatesVisual Arts 2013
Performance Task
Generalizability
Coefficient
Index of Dependability
1 (Compare and
Contrast)0.88 0.88
2 (Drawing)
0.74 0.74
Annual technical report provided to SC Arts Education Associate and posted
online each fall.
Full report available at: https://scaap.ed.sc.edu
REPORTING
https://scaap.ed.sc.edu
Report Cards
Revisions to report cards made based on teacher feedback
Report cards generated in collaboration with programmer
Multiple-choice section of report cards generated and disseminated prior to end of school year in May
Full report cards including performance tasks results disseminated the following September.
Sample Report Card
Research on the SCAAP Assessments
Comparing the Dimensionality Structures of Music & Visual Arts Multiple-Choice Assessments (SCEPUR, 2006)
An Exploratory Study of the Dimensionality Structure of a Music Multiple-Choice Assessment (AERA, 2006)
Efficacy of a Web-Based Training and Monitoring Procedure in Scoring Performance Tasks (AERA, 2007)
Raters Characteristics and Performance Scores (AERA, 2008)
Research on the SCAAP Assessments
Rhythm Syllable System and Rhythm Achievement (AERA 2008)
The Effect of Gender on a Language-related Theatre Task (SCEPUR, 2009)
Teachers’ Use of Assessment Results (AEA, 2010)
Teachers Making Meaning of Displays of Student Results (AEA, 2011)
Research Using SCAAP Results
Comparing Arts Achievement to English Language Arts and Mathematics Achievement in Arts Education Reform Schools (SCEPUR, 2005)
Evaluating the Program Characteristics of Arts Schools with Disparate Achievement Levels (SCEPUR, 2006)
Multiyear Evaluation of the Arts Education Reform Effort in South Carolina (AERA, 2007)
Investigating Arts Programs and Implementation Strategies for Infusing Arts Into Curriculum (AERA, 2007)
SCAAP Publications
Featured in an assessment textbook: Assessing Performance: Designing, Scoring,
and Validating Performance Tasks (Johnson, Penny, & Gordon, 2008)
Music Assessment Symposium Proceedings:
Assessment in Music Education: Integrating Curriculum, Theory, and Practice (Yap & Pearsall, 2007)
