Open Source Innovationsfor Better Assessment

Brandt ReddCTO

NCSA22 June 2015

2

Smarter Balanced at a Glance

• 23 Members: 21 States, US Virgin Islands, Bureau of Indian Education

• Represent more than 14 million students.

• Formative, Interim, and Summative assessments aligned to the Common Core State Standards for English Language Arts andMathematics.

3

Achievements in the Last 12 Months

• New home at UCLA

• Released open source assessment software (Assessment Production, Test Delivery, Scoring, Reporting, Digital Library)

• Summative tests delivered to 6.5 million students in 18 states.

• Millions of interim tests delivered.

• More than 570,000 registered teachers on the digital library.

• Nearly 200 colleges accept Smarter Balanced assessment for placement purposes with more on the way.

4

Devices Used in Spring 2015 Tests

Windows47%

ChromeBook33%

MacOS12%

iPad8%

Others1%

Fraction of 38 million test sessions using each device type.

5.6% Windows 838.5% Windows 7 0.1% Windows Vista 2.3% Windows XP(percent of all devices)

(Preliminary Results as of 22 May 2015)

5

Innovative Features

• Constructed Responses: Graphing, Equations

• Accessibility Features: Text-To-Speech, Refreshable Braille, Tactile Graphics, more

• Computer Adaptive Tests (CAT)

6

SmarterApp.orgSpecifications

• Core Standards (Format and API)• Item Format• Item Metadata• Content and Test Packaging• Test Results• Reporting API• Data Extracts• More

7

Interoperability Standards and Specifications used by SmarterApp

• Common Core State Standards

• SmarterApp Assessment Item Format

• Common Education Data Standards (CEDS)– Smarter Balanced Test Results Loaded into “Align”

• IMS Content Packaging

• SAML and OAuth (Single sign-on)

• Smarter Balanced Profile of IMS QTI/APIP*

• LRMI and Learning Registry*

• Accessible Rendered Item format (ARI)**Planned

Interoperability Challenge

• Each member state chooses its service provider to administer, score, and report the Smarter Balanced tests.

8

9

Technical Challenges

• Deploying open source across vendors• Packaging assessments for multiple

vendors• Collecting data in multiple formats from

multiple states and vendors• Efficient assessment production system• Sustaining pace of innovation

10

Open Source Tools for theAssessment Lifecycle

TestBlueprintStandards

ItemSpecifica-

tions

ItemAuthoring

ItemBanking

TestAuthoring

TestPackage

TestSpecifica-

tions

TestDelivery

ItemScoring

TestScoring

TestResults

HandScoring

ReportingIndividual

& AggregateReports

DataWarehouse

AssessmentDesign

AssessmentProduction

AssessmentDelivery

Reporting& Analytics

11

Assessment Production

• Current Status– Contracted out for Assessment Production Support in

2014-2015 and 2015-2016– Open Source Item Authoring, Item Bank, and Test

Authoring tools on SmarterApp.org

• Roadmap– Integrated assessment authoring system based on

software production systems– Software RCS for Item Pool– Open Source and Multivendor Item Authoring– Open Source Test Authoring and CAT Simulation

12

Assessment Delivery

• Current Status– Summative tests delivered to more than 6.5 Million Students in

Spring 2015.– 18 States and 5 Test Delivery Vendors– 3 States used the Open Source Test Delivery System (TDS)

• Enhancements for 2015-2016– Increased robustness, convenience, and capacity in the TDS– Universal Secure Browsers– Accessibility Updates

• Enhancements for 2016-2017– Horizontal server scaling– New Accessible Rendered Item (ARI) format– Summative Handscoring– Graphing Calculator– Accessibility Updates

13

Reporting & Data Warehouse

• Current Status– Used by 4 States in 2014-2015– Some hosted by Smarter balanced, others hosted by

vendors

• Roadmap– Updated “Test Results Transmission” format– Public Reporting– Clearer and more customizable Individual Student

Reports– Links to the Digital Library

14

Digital Library of Formative Materials

• Current Status– More than 2,200 curated resources– More than 575,000 registered teachers– Community features – content submission,

sharing of experiences, feedback

• Roadmap– Better integration with the Learning Registry– Improved Search– Local Community Groups– More Content– More Community Features

15

Accessible Rendered Item Format

• “Like a PDF for Assessment Items”

• Facilitates innovative new item types.

• Complimentary to QTI/APIP

• Leverages JavaScript for accessibility and scoring

• Supports multiple item authoring systems – each specializing in certain item types

• Rendering modes:– Response Capture– Inline feedback (for formative use)– Response rendering (for scoring and review)

• xAPI and Caliper compatibility

16

Opportunities

• For Assessment Technology Vendors– Watch for Smarter Balanced RFPs– Incorporate SmarterApp technology into your

solutions for interoperability and cost savings– Deploy technology for Smarter Balanced states

• For Assessment Developers– Leverage production and delivery tools– Consistent delivery platform across subject areas

• For States and Districts– Consistent secure browser technology across all

assessments– Build state and local assessments on the same

technology platform as Smarter Balanced

17

Q&A

• Resources– http://www.smarterbalanced.org– http://www.smarterapp.org

• Specifications including test results

Sample Individual Student Report

18