View
213
Download
0
Category
Tags:
Preview:
Citation preview
26th Annual Management Information Systems [MIS]
ConferenceFebruary 14, 2013Washington, DC
Common Education Data Standards (CEDS) Supporting
Assessment Systems Development
Copyright © IMS Global and SIF Association
Overview• Need• What is AIF• Status• Demonstration Prototypes• Discussion and Questions
Copyright © IMS Global and SIF Association
Created by Wordle.netCopyright © IMS Global and
SIF Association
Need• Standards– Confusing
• Assessment– RTTA and Consortia– Formative– Accountability
• Inform InstructionStock photo.
Assessment Lifecycle
ContentDevelopment
Pre-TestAdministration
TestAdministration
Scoring
Reporting
Post-TestAdministration
• Content and data reviews• Test form construction• Field testing• Item banking & statistics• Content exchange /
interoperability
•Planning & blueprinting•Item types• Content development &
universal design• Learning standard
alignment
• Administration planning & scheduling
• Registration, assignment,• Form sampling• Online infrastructure
readiness assessment• Pre-session planning
(paper / online) & setup• Alternate form
assignment
• Test form delivery• Platform (paper, online,
mobile) presentation• Item content & tools• Adaptive testing• Response collection• Proctoring controls• Form content security• Desktop security• Accessibility• Testing anomalies
• Computer scoring• Professional scoring• Algorithmic (AI) scoring• Portfolio scoring• Sub test / strand scoring
• Attemptedness• Performance levels• Scaling / norming• Growth scores• Range finding
AssessmentLife Cycle• Individual reporting
• Diagnostic reporting• Informing & personalizing
instruction• Performance on
standards• Dashboard / summary
reporting• Aggregation / disaggregation• Exchanging results / data
• Psychometric analysis• Equating• Score tables - scaling, norming• Performance levels / cut
scores• Field test analysis• Aligning results with
curriculum / instruction• Program and teacher
effectiveness
What is AIF?
• Systemic view of an assessment system and applications
• Demonstrates points of interoperability between applications within system
• Outlines the data model and transport
Copyright © IMS Global and SIF Association
REGIONAL Systems
Assessment Platform Main Components
STATE Systems
Assessment Creation & Management System(ACMS)
LOCAL Systems
SIF
Assessment Delivery System(ADS)
Assessment Reporting System
(ARS)
APIPAPIP
SIF
APIP
SIF
APIP
Assessment Score Processing System
(ASPS)?
APIP
USDoE / EDFacts
Asse
ssm
ent P
latf
orm
Sub
-Com
pone
nts
REGION
STATE / ConsortiaSLDS, Accountability/AYP, Growth
Assessment Registration &
Administration System (ARAS)
Assessment Creation & Management System(ACMS)
LOCAL (LEA/School)SIS, LMS, Gradebook, Reporting System
AssessmentData
Warehouse (ADW)
3
2
6
9
1
AssessmentResults
OperationalReporting
(AROR)
10
11
4
Assessment Presentation &
Session Management System (APSMS)
Assessment Delivery System (ADS)
Assessment Reporting System (ARS)
16
Test Banking
Item Banking
PNP
PNP
Assessment Scoring Management
System (ASMS)
Assessment Analytics
System (AAS)
17 18 19
13
15 14
12
Assessment Scoring Analytics System
(ASAS)
20Assessment ScoreProcessing System (ASPS)
8
Adaptive Engine
21
Learning Standards 2222
Status
• Use cases outlined• Elements complete• Public review complete• Prototype demonstrations complete• Best practice document complete• Available on ceds.ed.gov/aif.aspx
Demonstration Pilots
• Logical and Physical Testing• Arrow 1• Arrow 10• Arrow 14
Arrow 1• Item Bank to Item Bank
USDoE / EDFacts
Ass
ess
me
nt P
latf
orm
Su
b-C
om
po
ne
nts
REGION
STATE / ConsortiaSLDS, Accountability/AYP, Growth
Assessment Registration &
Administration System (ARAS)
Assessment Creation & Management System(ACMS)
LOCAL (LEA/School)SIS, LMS, Gradebook, Reporting System
AssessmentData
Warehouse (ADW)
3
2
6
9
1
AssessmentResults
OperationalReporting
(AROR)
10
11
4
Assessment Presentation &
Session Management System (APSMS)
Assessment Delivery System (ADS)
Assessment Reporting System (ARS)
16
Test Banking
Item Banking
PNP
PNP
Assessment Scoring Management
System (ASMS)
Assessment Analytics
System (AAS)
17 18 19
13
15 14
12
Assessment Scoring Analytics System
(ASAS)
20Assessment ScoreProcessing System (ASPS)
8
Adaptive Engine
21
Learning Standards 2222
Arrow 1 Diagram
Assessment Reporting System (ARS)
Assessment Score Processing System (ASPS)
Assessment Delivery System (ADS)
Assessment Registration & Administration System (ARAS)
Assessment Creation & Management System (ACMS)
Assessment Data Warehouse (ADW)
Assessment Results Operational Reporting (AROR)
Assessment Presentation & Session Management System (APSMS)
Test Banking
Item Banking
Assessment Scoring Management System (ASMS)
Assessment Analytics System (AAS)
Assessment Scoring Analytics System (ASAS)
10 – Item Bank Interoperability
Consortia Item Banks
State Item Banks
Local Item Banks
Item Bank Admin
Author orPsychometrician
Arrow 1 Test ScenarioScenario Description• The purpose of this scenario is to demonstrate assessment
items moving from an item authoring system to an item bank or from one item bank to another. This scenario covers only assessment items moving from an item authoring system to an item bank.
• A consortium has secured item authoring system A. This Item authoring system will not house the items; it is simply a means for item creation. The items will reside in item banking system B. Item authoring system A has created the assessment items and the editing and revision cycle has been complete. Item authoring system A is ready to send the items to item banking system B for storage and assessment instrument creation.
Copyright © IMS Global and SIF Association
Test Scenario Components Required• Item authoring system• Item banking system
Pre-Condition• Assessment content (items, instruments, etc.) is ready to be
transferred in the sending item bank.
Post-Conditions• Content is now ready for use including reviews, edit,
extensions, assessment instrument creation, passing of instruments to delivery system, etc. Copyright © IMS Global and
SIF Association
Arrow 1 Test Scenario Cont’d
Copyright © IMS Global and SIF Association
Arrow 1 Process Flow
Arrow 10• Assessment Registration
USDoE / EDFacts
Ass
ess
me
nt P
latf
orm
Su
b-C
om
po
ne
nts
REGION
STATE / ConsortiaSLDS, Accountability/AYP, Growth
Assessment Registration &
Administration System (ARAS)
Assessment Creation & Management System(ACMS)
LOCAL (LEA/School)SIS, LMS, Gradebook, Reporting System
AssessmentData
Warehouse (ADW)
3
2
6
9
1
AssessmentResults
OperationalReporting
(AROR)
10
11
4
Assessment Presentation &
Session Management System (APSMS)
Assessment Delivery System (ADS)
Assessment Reporting System (ARS)
16
Test Banking
Item Banking
PNP
PNP
Assessment Scoring Management
System (ASMS)
Assessment Analytics
System (AAS)
17 18 19
13
15 14
12
Assessment Scoring Analytics System
(ASAS)
20Assessment ScoreProcessing System (ASPS)
8
Adaptive Engine
21
Learning Standards 2222
Copyright © IMS Global and SIF Association
Arrow 10 Diagram
Assessment Reporting System (ARS)
Assessment Score Processing System (ASPS)
Assessment Delivery System (ADS)
Assessment Registration & Administration System (ARAS)
Assessment Creation & Management System (ACMS)
Assessment Data Warehouse (ADW)
Assessment Results Operational Reporting (AROR)
Assessment Presentation & Session Management System (APSMS)
Test Banking
Item Banking
Assessment Scoring Management System (ASMS)
Assessment Analytics System (AAS)
Assessment Scoring Analytics System (ASAS)
20 - Registration
Local SIS or LMS
State SLDS or Info System
Test Coordinator orTeacher
State Data Admin
Local Data Admin orTeacher
Consortia Info Systems
Consortia Data Admin
Copyright © IMS Global and SIF Association
Arrow 10 Test ScenarioScenario Description• This scenario describes the necessary components for an assessment
registration. This scenario covers student demographic, teacher, hierarchy and PNP information.
• A consortium is ready to give an assessment. The student, school, LEA, PNP and teacher information have already been entered into the SIS or data warehouse. Assessment registration and administration system A pulls over all of the necessary information from the SIS or data warehouse B. The students are then assigned to a specific administration of an assessment. The registration and administration information is then passed from the assessment registration and administration system A to the assessment delivery system C.
• An alternate scenario would be that the PNP information is entered into the registration system after the information has been pulled over from the SIS or data warehouse.
Copyright © IMS Global and SIF Association
Arrow 10 Test Scenario Cont’dTest Scenario Components Required• Student information system or data warehouse that houses all of
the necessary information for registration• Assessment registration and administration system• Assessment delivery system
Pre-Condition• All of the student, school, LEA and teacher information have been
entered into the SIS and/or Data Warehouse.• An administration of an assessment(s) has been identified.
Post-Conditions• The students are registered for a specific administration(s) of an
assessment.Copyright © IMS Global and
SIF Association
Arrow 10 Process Flow
Copyright © IMS Global and SIF Association
Arrow 14• Reporting System to Data Warehouse
USDoE / EDFacts
Ass
ess
me
nt P
latf
orm
Su
b-C
om
po
ne
nts
REGION
STATE / ConsortiaSLDS, Accountability/AYP, Growth
Assessment Registration &
Administration System (ARAS)
Assessment Creation & Management System(ACMS)
LOCAL (LEA/School)SIS, LMS, Gradebook, Reporting System
AssessmentData
Warehouse (ADW)
3
2
6
9
1
AssessmentResults
OperationalReporting
(AROR)
10
11
4
Assessment Presentation &
Session Management System (APSMS)
Assessment Delivery System (ADS)
Assessment Reporting System (ARS)
16
Test Banking
Item Banking
PNP
PNP
Assessment Scoring Management
System (ASMS)
Assessment Analytics
System (AAS)
17 18 19
13
15 14
12
Assessment Scoring Analytics System
(ASAS)
20Assessment ScoreProcessing System (ASPS)
8
Adaptive Engine
21
Learning Standards 2222
Copyright © IMS Global and SIF Association
Arrow 14 Diagram
Assessment Reporting System (ARS)
Assessment Score Processing System (ASPS)
Assessment Delivery System (ADS)
Assessment Registration & Administration System (ARAS)
Assessment Creation & Management System (ACMS)
Assessment Data Warehouse (ADW)
Assessment Results Operational Reporting (AROR)
Assessment Presentation & Session Management System (APSMS)
Test Banking
Item Banking
Assessment Scoring Management System (ASMS)
Assessment Analytics System (AAS)
Assessment Scoring Analytics System (ASAS)
40 – Distribute Results (local)
50 – Distribute Results (state)
Local SIS or LMS
State SLDS or Info System State Data Admin
Local Data Admin orTeacher
120 – Distribute Summary (local)
130 – Distribute Summary (State)
LEA Information or Reporting System
LEA Data Admin
Consortia Information or Reporting System
Consortia Data Admin
Copyright © IMS Global and SIF Association
Arrow 14 Test Scenario
Scenario Description• The purpose of this scenario is to demonstrate sending
assessment results to the SEA from the Consortium administered assessment.
• The consortium has administered an assessment. The assessment results system A has received and compiled all information based upon the assessment. Assessment results system A compile, packages and disseminates the information to the SEA data warehouse B.
Copyright © IMS Global and SIF Association
Arrow 14 Test Scenario Cont’dTest Scenario Components Required• Assessment results system• State data warehouse or reporting system
Pre-Condition• Assessment results are available and have been collected so
that summaries can be produced.
Post-Conditions• The state system has the results loaded.
Copyright © IMS Global and SIF Association
Arrow 14 Process Flow
Copyright © IMS Global and SIF Association
Data Available by Entity• Assessment Family• Assessment• Assessment Form• Assessment Form Section• Assessment Form Subtest• Assessment Subtest Result• Assessment Performance Level• Assessment Item• Assessment Item Possible
Response• Assessment Item Response• Assessment Item Rubric• Assessment Participant Session• Learner Action• Assessment Administration• Assessment Registration• Assessment Session• Organization• Registration Accommodation• Person
• Achievement Evidence• Achievement• Learning Goal• Learning Assignment Learning
Resource• Learning Resource• Learner Activity• Learning Standard Document• Learning Standard Item• Competency Item_Competency Set• Competency Set• Learning Standard Item Grade Level• Assessment Levels for Which Designed• Assessment Form Subtest Levels for
Which Designed
Copyright © IMS Global and SIF Association
Documentation
• Assessment Interoperability Framework Definitions and Requirements
• Use Cases for the Assessment Interoperability Framework
• Data Elements• Best Practices• Demonstration Prototype
Copyright © IMS Global and SIF Association
Work Moving Forward
• Continue to evolve the technical standard within IMS and SIF respectively and reflect back to CEDS
• Implementation• Additional pilots• API and transport• Certification
Copyright © IMS Global and SIF Association
ContactJill Abbott, CEOAbbott Advisor Groupjabbott@abbottadvisorgroup.com
Rob Abel, CEOIMS Global Consortiumrabel@imsglobal.org
Larry Fruth, CEOSIF Associationlfruth@sifassociation.org
All current documentation can be found at http://ceds.ed.gov/aif.aspx - Assessment Interoperability Framework (AIF)
Recommended