View
34
Download
1
Category
Tags:
Preview:
DESCRIPTION
Operational CBT Implementation Issues: Making It Happen. Richard M. Luecht, PhD Educational Research Methodology University of North Carolina at Greensboro. - PowerPoint PPT Presentation
Citation preview
2010 R.M. Luecht 1
Operational CBT Implementation Issues:
Making It HappenRichard M. Luecht, PhD
Educational Research MethodologyUniversity of North Carolina at
GreensboroTenth Annual Maryland Assessment Conference: COMPUTERS AND THEIR IMPACT ON STATE ASSESSMENT: RECENT HISTORY AND PREDICTIONS FOR THE FUTURE. 18-19 October, College Park MD
2010 R.M. Luecht 2
What do you get if you combine a psychometrician, a test development specialist, a computer hardware engineer, a LSI software engineer, a human factors engineer, a QC expert, and a cognitive psychologist?
A pretty useful individual to havearound if you’re implementing CBT!!!!
2010 R.M. Luecht 3
A Naïve View of Operational CBT*
* Includes linear CBT, CAT, CMT CAST and other variants
ku,,uUjk Rj:ˆImaxikiij
11
Item Selection/Test Assembly Algorithm
,:u,,ugmaxˆkkii ii
MAPuu 1111
Ability Estimation/Scoring Algorithm
Item #001Item #001
Item #001Item #001
Item Bank
Ui=010120113Response Vector
Test Delivery Network
Ethernet
Server
Data
Data Server
Data
Examinee
2010 R.M. Luecht 4
The Challenge of CBTMoving more complex data more quickly, more securely and more accurately from item generation through final scoringImmediate responsiveness where possibleRe-engineering data management and processing systems: end-to-end99.999999% accuracy and eliminating costly and error-prone human factors through automation and better QC/QA
2010 R.M. Luecht 5
Systems Impacted by Redesign and QC/QA
Item development and bankingTest assembly and compositionExaminee eligibility, registration, and scheduling, feesTest deliveryPsychometrics and post-examination processing
Item analysis, key validation and quality assuranceTest analysisFinal scoring, reporting and communication
2010 R.M. Luecht 6
Item and Item Set Repositories
2010 R.M. Luecht 7
Test Data Repositories
Test UnitRecords
Test Unit IdentificationItem List ReferenceTiming DataNavigation ControlsPresentation ScriptsTemplate Reference
ResourceVersions
Version/ReleaseTest Units ReferenceEncryption Key(s)Restructured Test Units(e.g., XML)
Item Index
Test Unit IdentificationItem Identification
Test Repository CBT Resource Files
2010 R.M. Luecht 8
Examinee Data Repositories
ExamineeRecords
Examinee IDContact InformationDemographics
ResultsRecords
Examinee IDTest IdentificationItem-Level TransactionBLOB
Test/ScoreHistory
Examinee IDTest ReferenceTesting DateScore(s)Accommodations
Examinee Repository CBT Server Repository
2010 R.M. Luecht 9
CBT: A System of Systems
2010 R.M. Luecht 10
Item-Level DataItem or exercise rendering data
Stimulus information (e.g., MCQ stem, a reading passage)Response display labels (e.g., distractors as labels for a check box control)Scripts for interactivityTemplate references
Content and other item attributesContent category codesCognitive and other secondary classificationsLinguistic features
2010 R.M. Luecht 11
Item-Level Data – cont.Statistical item data
Classical item statistics (p-values, biserial correlations, etc.)IRT statistics (1PL, 2PL, 3PL, GPCM parameter estimates)DIF statistics and other special indices
Operational dataReuse historyExposure rates and controls (for CAT)Equating status
2010 R.M. Luecht 12
Test Unit DataObject list to include (e.g., item identifies for all items in the test unit)Navigation functions, including presentation, review and sequencing rulesEmbedded adaptive mechanisms (score + selection)Timing controls and other information (e.g., how clock functions, time limit, etc.)Title and instruction screens
2010 R.M. Luecht 13
Test Unit Data – cont.Presentation template references
Helm look-and-feel (navigation style, etc.)Functions (e.g., direction of cursor movement after ◄┘ or tab is pressed)
Reference and ancillary look-up materialsCalculatorsHyperlink to other BLOBs Custom exhibits available to test takers
2010 R.M. Luecht 14
Standard Hierarchical View of a “Test Form”
Test Form A
Section IISection I Section III
Group 1 Group 2Set 1
Item1
Item2
Item3
Item5
Item6
Item7
Item4
Item8
Item9
Item10
Item12
Item11
2010 R.M. Luecht 15
Examinee DataIdentification information
Name and identification numbersPhoto, digital signature, retinal scan informationAddress and other contact information
Demographic informationEligibility-to-test information
JurisdictionEligibility periodRetest restrictions
2010 R.M. Luecht 16
Examinee Data – cont.
Scheduled test date(s)Special accommodations requiredScores and score reporting informationTesting history and exam blockingSecurity history (e.g., previous irregular behaviors, flagged for cheating, indeterminate scores, or large score gains)General correspondence
2010 R.M. Luecht 17
Interactions of Examinee and Items or Test Units
Primary informationFinal responsesCaptured actions/inactions (state and sequencing of actions)
Secondary informationCumulative elapsed time on “unit”Notes, marks or other captured during testing
2010 R.M. Luecht 18
Response Processing in CBTResponse capturing agents convert examinee responses or actions to storable data representationsExamples
item.checkbox.state (T/F) item.response.choice=“A” {A,B,C,D}item.group.unit(j).state (T/F), j=1,…6 item.response.choice=“1,4” {1,2,3,4,5,6}item.component.text(selected=position,length) item.response.text=“text”item.component.container(freeresponse.entry) item.response.text=“text”case.grid.cellRowCol(numeric.entry) case.grid.cellref.response.text=“value”
2010 R.M. Luecht 19
Raw Response Representations for Discrete
ItemsUniform CPA Examination: Item 002
Item 1
Item 2
item002.checkbox1.state="false"
item002.stemtext.text="HTML formatted text"
item002.checkbox2.state="false"
item002.checkbox3.state="true"
item002.checkbox4.state="false"
item002.checkbox5.state="false"
Convert to item002.response.choice=“3”
2010 R.M. Luecht 20
Raw Response Representations for a Performance ExerciseCPA Examination: Simulation A
Item 1
Item 2
Item 3
Case AItem 4Item 5
HelpContext Reports12-18-01 Instructions
Date Account
12-19-01
12-19-01
12-19-01
12-19-01
12-19-01
12-19-01
280-04
345-02
280-01
345-02
280-04
Debit
12501.99
Credit
C4C3C2
R7
R6
R5
R4
R3
R2
R1
C1
280-01
Convert to…caseA.tab5.sheet001.r3c4.text=“12501.99”
2010 R.M. Luecht 21
Raw Response Representations for an
EssayStore as…RichTextBox.Item001.text=“There were two important changes that characterized the industrial revolution. First, individuals migrated from rural to urban settings in order to work at new factories and in other industrial settings (geographic change). Second, companies began adopting mechanisms to facilitate mass production (changes in manufacturing procedures, away). ”
2010 R.M. Luecht 22
Entering the Psychometric Zone: Data Components of Scoring
EvaluatorsResponsesSelections, actions or inactions: item.response.state=control.state (ON or OFF)Entries: item.response.value=control.value
Answer expressions (rethinking IA is needed)
Answer keysRubrics of idealized responses or patterns of responsesFunctions of other responses
Score evaluators process the responsesScoring evaluators convert the stored responses to numerical values—e.g., f(responseij, answer keyi)xij[0,1]Raw scoring or IRT scoringaggregation and scaling of item-level numerical scores
2010 R.M. Luecht 23
Planning for Painless Data Exchanges and ConversionsSystems and subsystems need to exchange data on a regular basis, providing different views and field conversionsThe hand-off must have several fool-proof QC steps
Verification of all inputsConversion success 100% verifiedReconciliation of all results, including counts, discrepancies, missing values, etc.
2010 R.M. Luecht 24
Example of a (Partial) Examinee’s Test Results
Recordtestp>wang>marcus>>605533641>0A1CD9>93bw100175>1>>501>001>ENU>CB1_CAST105>>90>0>0>0>DTW>06/26/96>08:41:38>05:58:32>w10>2>apt 75>1000 soldiers field rd>north fayette>IN>47900>USA>1>1235552021>>NOCOMPANYNAME>0>>>>>1>1235551378>>0>0>35>>142/218/0/u>1>-1>7>CBSectionI.12>CB1>s>p>0>36>>72/108/0/u>Survey015>survey15>s>p>0>0>>0/0/0/u>CBSectionI>CAST2S1>s>p>0>28>>28/62/0/u>CBSectionII>CAST2S4>s>p>0>42>>42/48/0/u>CBSectionII>CAST2S3>s>p>0>0>>0/0/0/u>CBSectionII>CAST2S2>s>p>0>0>>0/0/0/u>Survey016>survey2>s>p>0>0>>0/0/0/u>0>372>SAFM0377>2>0>E>5>s>E>1>76>>SAEB0549>2>0>D>5>s>A>0>68>>SAFM0378>2>0>A>5>s>A>1>72>>SAAB1653>2>0>C>5>s>D>0>102>>SABA8868>2>0>B>5>s>C>0>85>>SCAA1388>2>0>E>8>s>E>1>53>>SAAA8447>2>0>D>5>s>E>0>55>>SAAB1934>2>0>A>5>s>A>1>60>>SAAB2075>2>0>E>5>s>E>1>136>>SADA7710>2>0>D>5>s>D>1>40>>SABB1040>2>0>B>5>s>E>0>46>>SCAA1396>2>0>H>10>s>A>0>93>>SACA8906>2>0>D>5>s>E>0>75>>SADA8116>2>0>C>5>s>D>0>53>>SADA8673>2>0>B>5>s>B>1>41>>SACA8626>2>0>B>5>s>D>0>48>>SAFM0374>2>0>C>5>s>D>0>80>>SABA6397>2>0>A>5>s>A>1>110>>SAAB1088>2>0>C>5>s>C>1>55>>SACA8455>2>0>D>4>s>D>1>73>>SAAB1667>2>0>C>5>s>C>1>44>>SAAJ7633>2>0>C>5>s>C>1>89>>SABA5745>2>0>D>5>s>A>0>43>>SCAA1389>2>0>B>8>s>H>0>61>>SADA8650>2>0>A>5>s>C>0>39>>SAFB0112>2>0>C>5>s>C>1>132>>SAAB2513>2>0>B>5>s>B>1>120>>SAFA9248>2>0>E>5>s>A>0>77>>SABJ1042>2>0>D>5>s>C>0>112>>SACJ5894>2>0>C>5>s>D>0>82>>SAAA0410>2>0>D>5>s>E>0>89>>SAAB1681>2>0>C>5>s>C>1>88>>SAFM0365>2>0>A>5>s>A>1>65>>SAEA8980>2>0>A>5>s>B>0>52>>
2010 R.M. Luecht 25
Assessment XML<?xml version="1.0" encoding="UTF-8"?><AssessmentResult xmlns="http://ns.hr-xml.org/2004-08-02" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ns.hr-xml.org/2004-08-02AssessmentResult.xsd">
<ClientId idOwner="Provider Inc"><IdValue name="ClientCode">OurClient-1342</IdValue>
</ClientId><ProviderId idOwner="Customer Inc">
<IdValue>ePredix</IdValue></ProviderId><ClientOrderId>
<IdValue name="PO Number">53RR20031618</IdValue><IdValue name="Department Name">Administration</IdValue>
</ClientOrderId><Results>
<Profile>Customer Service</Profile><OverallResult>
<Description>Executive Manager</Description><Score type="raw score">51</Score><Score type="percentile">65</Score><Scale>40-60</Scale>
</OverallResult><AssessmentStatus>
<Status>In Progress</Status><Details>Remains: "GAAP Basic Knowledge"</Details><StatusDate>2003-04-05</StatusDate>
</AssessmentStatus><UserArea/>
</AssessmentResult>
2010 R.M. Luecht 26
Translating XML Entities to a Data Structure
Application
ContentHandler
ErrorHandler
DTD andSchemaHandler
EnitityResolver
ExternalDataEntity
ExternalDataEntity
XMLDocument
External DTDSchema or XSL
Sheets
XML Parser StructuredData
2010 R.M. Luecht 27
Extracting Data ViewsA data view is a set of restructuring functions that produce a data set from raw data
Views begin with a queryUsually results a formatted file structureGraphing functions produce graphic data setsDatabase functions produce database record sets
Multiple views are possible for different uses (e.g., test assembly, item analysis, calibrations, scoring)Well-designed views are reusable
Standardized queries of the database(s)Each views as a template with “object” statusViews can be manipulated by changing their properties (e.g., data types, presentation formats)
2010 R.M. Luecht 28
Types of Data Files (Views)Implicit Files: File format implies a structure for the data
Flat files with fixed columns (headers optional)Comma, tab or other delimited files
Explicit Files: Variables, data types, formats and the actual data are explicitly structured
Data base files: dBASE, Oracle, Access, etc.Row-column worksheets with “variable sets” (e.g., Excel in data mode, SPSS)XML and SGML
2010 R.M. Luecht 29
Explicit Structured “List View”
2010 R.M. Luecht 30
Structure of the Flat File(A Type of “Metadata”)
Data Type Definitions
PresentationView Data
2010 R.M. Luecht 31
Relational DBM in Access
2010 R.M. Luecht 32
Relational Schema Across Databases for ATA
Item Attribute Table
PK Item ID Code
Item NameClassification SchemeAttibute Code
Classifications
PK Classification Scheme
Classification Description
Attribute Table
PK Attribute Code
Attribute DescriptionFK1 Classification Scheme
Classification Attribute Constraints
PK,FK1 Attribute Code
Classification SchemeConstraint Expression
Eligible Item Table
PK,FK1 Item ID Code
Item IDAnswer Key TypeAnswer Key Expression CountScore CategoriesItem MeanItem SDPt. BiserialD ConstantA ParameterB ParameterC ParameterWord CountReadability
2010 R.M. Luecht 33
Data File Extractions for Psychometric Processing
Item Analysis and Key Validation, Calibration and Equating
Table.Active_Test_Forms Table.Active_Test_Form_Content
Table.Link_Item_Sets
Table.Active_Item_Bank_Keyed
Table.Active_Examinee_Test_Forms
Table.Active_Examinee_Responses
2010 R.M. Luecht 34
Query Test Form and Item Databases
SELECT Item.Records IF(Query_Conditions=TRUE)Test.TestID Item.ID
TST0181 ITM020920TST0181 ITM048392TST0181 ITM020342TST0181 ITM038632TST0181 ITM023833TST0181 ITM031935TST0181 ITM035329TST0181 ITM022222TST0183 ITM020921TST0183 ITM030102TST0183 ITM028902TST0183 ITM038886TST0183 ITM024519TST0183 ITM039981TST0183 ITM040027TST0183 ITM030253
Test.TestID Item.IDTST0181 ITM020342TST0181 ITM020920TST0183 ITM020921TST0181 ITM022222TST0181 ITM023833TST0183 ITM024519TST0183 ITM028902TST0183 ITM030102TST0183 ITM030253TST0181 ITM031935TST0181 ITM035329TST0181 ITM038632TST0183 ITM038886TST0183 ITM039981TST0183 ITM040027TST0181 ITM048392
Sort Item.ID
2010 R.M. Luecht 35
The P I QuerySELECT Examinee.Records IF(Query_Conditions=TRUE)
Exam.PersonID Exam.TestID Exam.Status Exam.Date107555 TST0183 F 11-Sep-04517101 TST0181 F 11-Sep-04670048 TST0181 F 13-Sep-04758735 TST0183 F 13-Sep-04754364 TST0183 F 13-Sep-04827960 TST0183 R 13-Sep-04619834 TST0183 F 13-Sep-04615233 TST0182 R 13-Sep-04429336 TST0182 F 14-Sep-04
Exam.PersonID Exam.TestID Exam.ItemID Exam.iSequence Exam.iTime Exam.Response107555 TST0183 ITM020921 1 39 B107555 TST0183 ITM030102 2 131 D107555 TST0183 ITM028902 3 39 C107555 TST0183 ITM038886 4 61 B107555 TST0183 ITM024519 5 58 D107555 TST0183 ITM039981 6 67 A107555 TST0183 ITM040027 7 61 A107555 TST0183 ITM030253 8 34 B517101 TST0181 ITM020920 1 65 C
2010 R.M. Luecht 36
The P I QueryGENERATE.FLATFILE(Examinee.Records,Item.Records)
Examinee.PersonID ITM
0203
42
ITM
0209
20
ITM
0209
21
ITM
0222
22
ITM
0238
33
ITM
0245
19
ITM
0289
02
ITM
0301
02
ITM
0302
53
ITM
0319
35
ITM
0353
29
ITM
0386
32
ITM
0388
86
ITM
0399
81
ITM
0400
27
ITM
0483
92
107555 B D C D B B A A517101 B C A D C C B A
107555 9919911019991009517101 1190199990019991
.PersonID Items from TST0181 & TST0183
Generate“Masked” Response
File
2010 R.M. Luecht 37
Implied Flat File View of Test Data
(Person by Item Flat Files)Example 1: “raw response vectors” (input to commercial item analysis software)
00001 BDCAABCAEDACBD00002 BDBAABDAEDBCBD00003 BCCBABCAEDABBD
Example 2: “scored response vectors” (input to commercial item calibration software) 00001 11111111111011
00002 1101110111001100003 10101111111100
2010 R.M. Luecht 38
Reconciliation 101Definitions: bringing into harmony, aligning, balancingReconciliation is essential for CBT data management and quality assurance
Test forms, items, sets, examinees, and transaction output counts match input countsResults match expectations or predictions
2010 R.M. Luecht 39
Reconciliation Example (Examinee Data for
IA/Calibrations)File Reconcilation and Rectangular File Creator (R. Luecht, [c] 2009, 2010)Date/time: mm-dd-yy hh:mm:ssControl File: ControlFile.CONExaminee_Test_Form File: ActiveExamineeTestForm.txt NP = 1687
Treatment of (Score_Status=1) items: INCLUDE Items and ResponsesItem File: MasterItemFile.DAT NI = 3857, Total Read = 3857 Excluded= 0 ===========================================================Active_Examinee_Responses File=ActiveExamineeResponse.txt No. IDs (from Active_Examinee_Test_Form)= 1687 File size (examinee transactions)= 506100 No. nonblank records input = 506100 No. records with unmatched items = 0Forms = 8 1687 scored response records saved to Data-ResponseFile-Scored.RSP 1687 raw response records saved to Data-ResponseFile-Raw.RAW
ITEM LISTING and FORM ASSIGNMENTS DETECTED
ID Identifier NOpt Opts N-Count NFrm FormsItem_21801 5 ABCDE 41 1 04Item_24601 5 ABCDE 97 3 01 03 07Item_29801 5 ABCDE 97 2 02 07 :<only partial records included to conserve space>
MISMATCH SUMMARY----------------NO unmatched item IDs to FORMSNO unmatched item IDs to RESPONSE RECORDS
Form ID 01 02 03 04 05 06 07 08
01:SampleForm001 300 0 0 37 0 37 0 0
02:SampleForm002 0 300 0 0 200 0 0 0
03:SampleForm003 0 0 300 0 0 0 200 200
04:SampleForm004 37 0 0 300 0 200 0 0
05:SampleForm005 0 200 0 0 300 0 0 0
06:SampleForm006 37 0 0 200 0 300 0 0
07:SampleForm007 0 0 200 0 0 0 300 200
08:SampleForm008 0 0 200 0 0 0 200 300
Item Counts by Form
2010 R.M. Luecht 40
Follow the Single-Source Principle
A unique master record should exist for every entity
Examinees registered/eligible to testItemsItem setsModules, testlets or groupsTest forms
Changes should be made to the master and forward-propagated for all processing
2010 R.M. Luecht 41
Example of Single Source
NO!
YES!!
2010 R.M. Luecht 42
Ignorable Missing Data?Very little data is missing completely at random, limiting the legitimate use of imputationSome preventable causes of missing data
Lost records due to crashes/transmission errorsCorrupted response capturing/recordsPurposeful omitsRunning out of time/motivation to finish
2010 R.M. Luecht 43
Challenges of Real-Time Test Assembly (CAT or LOFT)
Real-time item selection requires high bandwidth and fast servers and pre-fetch reduces precisionA “test form” does not exist until the examination is complete
QC of test forms is very difficult, except by audit sampling and careful refinement of test specifications (objective functions/constraints)QC of the data against “known” test-form entities is NOT possible
2010 R.M. Luecht 44
Thanks!Ric Luecht
rmluecht@uncg.edu
Recommended