Progress Testing with SIRA Case Study

Based on the McMasterUndergraduate MD Programme

SIR UK User Group ConferenceAberdeen, UK, 21 June 2002

David Keane, Research Associate (keaned@mcmaster.ca)Programme for Educational Research and Development

Faculty of Health Sciences, McMaster UniversityHamilton, Ontario, Canada www.fhs.mcmaster.ca/perd

Objectives

1. Introduction to progress testing– definition– purpose– method– goals– special features– basic patterns in performance data

2. Using SIR for progress testing

Progress testing \ a definition

•longitudinal testing of knowledge acquisition

Progress testing \ a definition

•longitudinal testing of knowledge acquisition

•an objective method for assessing the acquisition and retention of knowledge over time relative to curriculum-wide goals

Progress testing \ definition detail

• objective– use multiple choice questions

• knowledge– test what learners know

• over time– test repeatedly at regular intervals

• curriculum-wide– address end-of-programme learning

objectives

an objective method for assessing the acquisition and retention of knowledge over time relative to curriculum-wide goals

Progress testing \ the purpose

•to determine whether the learner is progressing

– learning enough?

– retaining enough?

– doing so quickly enough?

Progress testing \ the method

•progress is relative– compare learner to his/her peer group

• current class or past n classes

• standardized scores (z-scores)

– review performance on multiple tests• current and past

•assessed with one measure– percentage correct, whole test

• adjust for guessing?

Progress testing \ goals

•help the learner (formative evaluation)

– constructive feedback• about their knowledge base

• about their ability to self-assess

• has to be specific/detailed

– timely feedback• reassure those who are progressing

• alert those who are not– do so early enough to facilitate effective

remediation

Progress testing \ goals

•help the Programme (summative evaluation)

– provide defensible evidence to support critical decisions pertaining to individuals

• mandated remediation

• pass / fail / conditional advancement

• the emphasis..– on formative aspects

• minimize negative impact on learning behaviours

– tutorial functioning– self/peer-assessment

Progress testing \ special features

•the item bank– a sample of the knowledge that a

good student will likely encounter by the time that student. .• graduates?

• is six months / a year beyond graduation?

- content encompasses nearly the 'entire' domain of the field in question

• cf. course/curriculum 'core' knowledge

Progress testing \ special features

• instructions to examinees

– don't study for this test• 180 items, randomly selected from 2,600+

– don't guess• test your ability to self-assess

• penalty for guessing (optional)

– attempt only those items for which you have some knowledge and are reasonably confident you know the best/correct answer

Basic patterns in performance data

• Class means on whole test for..– % attempted

– % correct• not adjusted for assumed guesses

• look at patterns..– across time

• week in programme (x of 138)

– across classes at week x

gif: Items Attempted (%)

gif: Items corect (%)

Basic patterns in performance data

• % attempted, % correct

– patterns are relatively stable across tests and classes

– means at week x are relatively consistent across tests and classes

– examinee performance is relatively consistent across tests and classes• overall test reliability 0.6 - 0.7 (8 tests)

• test-retest correlation 0.6 - 0.8 (2 tests)

End of Part I

Introduction to Progress Testing

Any questions?

Objectives

1. Introduce progress testing

2. Using SIR for progress testing– what's in an item?

– data management tasks

– managing dm tasks• software• databases and pql

– SIR \ valued features

– future enhancements

What's in an item? (1)

• the examinee sees..

nn. An elderly woman has been showing signs of forgetfulness, poor concentration, and decreased energy and appetite. On exami-nation her cognitive functioning seems quite good and her mini-mental (Folstein) score is 27/30. The most likely diagnosis is:

A Anxiety disorderB Multi-infarct dementiaC Alzheimer diseaseD Personality disorderE Depression

Stem

Options

What's in an item? (2)

• the data manager sees..– stem and options (text)

and

– unique item identifier

– correct response code– content codes (6 fields, 1, 2 or 3 sets)

– item performance data• stats on usage, power to discriminate• by test, class; across tests, classes

– and more..• date last used, don't use flag

Data management tasks

1. store, retrieve, manipulate and print

large volumes of textual information

• pre-test: test booklets– 180 items, 21–22 pages/booklet

• post-test: performance reports– for examinees: 2 reports x 1–2 pages/rep– for administrator: re. items, tests, classes and

examinees who are not progressing

• accommodate special needs re.– special characters – Greek letters, math symbols– page layout, fonts, typeface style

• merge data into report templates

Data management tasks \ post-test

2. read examinees' responses– 100-item optical mark response sheets

– tab-delimited ascii records• Mac: 2 sheets X approx. 280|420 examinees

3. score examinees' responses– requires item, test, class, examinee info

– compute and retain performance stats• key measures: % attempted, % correct

– mean & sd re. whole test (and major subdomains?)

• for: each examinee, each class|peer group

Data management tasks \ post-test

4. compute and retain item performance

stats– requires item, test, class, examinee info

5. compute/retrieve data needed in standard reports

– re. examinees, classes, tests, items

6. assemble and print reports

Data management tasks

7. enable support staff to do all of the above with relative ease

– minimal reliance on the application programmer after everything is up and running

What's the best tool for the job?

• SIR is not a word processor

• SIR is a record management and stats-oriented reporting tool– allows user to build powerful custom

applications

– vendor provides exceptional support beyond the installed Help files• prompt, relevant and practical replies

Solutions \ the best tool

• the MD Programme's solution

– for text-intensive tasks..

Corel WordPerfect

- for numeric data and stats-intensive tasks..

SIR

Solutions \ Corel WordPerfect

• a set of merge data files (database)• case-based by item id• item stems, options and other fixed info

• export data via csv or fixed-format records

• import data via csv-format records• into merge data files

• multiple merge forms (report templates)

• extensive use of merge and macro commands to produce pre/post-test reports

• custom-build merge|macro applications

Solutions \ SIR ver. 3.2 - 2000

• 2 databases, case-based – ITEMS re. items– TEEX re. tests, examinees,

classes

• major reliance on (vb) PQL– custom applications

• csv-format records– add/update records/fields (eg, from WP)

– write records/values (eg, for WP)

– PQL procedures• csv save, tabulate, save table, spss save

SIR 2000 \ valued features (1)

• DBMS

– case-based option for db type• system-maintained• easy access to any case's records• case id is on all dumped records

– global variables• pass user settings to applications

– utilities• Data \ File Dump, File Input

– tabfiles and tables• create, index, populate, delete tables

SIR 2000 \ valued features (2)

• PQL– nested access to cases– read csv-format records– vb dialog boxes

• PQL Procedures– write csv-format records– xtab tables, flexibility re. headers (columns)

– write SPSS system files

Future enhancements

• upgrade to SIR 2002 (from SIR 2000)

– update custom applications (to vb pql)

– add secondary indexes• examinees by name, current class

• web access– for examinees: performance reports

– method?• ColdFusion (CF –– SQL –– ODBC driver –– SIR db)

• CGI scripts

End of Part II

Using SIR for progress testing

Any questions?