MAPP and VSA The Role of the Measure of Academic Proficiency and Progress in the Voluntary System of Accountability Bill Wynne Solutions Implementation

MAPP and VSAThe Role of

the Measure of Academic Proficiency and Progressin the Voluntary System of Accountability

Bill WynneSolutions Implementation Specialist

ETS Higher Education Division

Confidential and Proprietary. Copyright ©2006 Educational Testing Service.

2

What are “outcomes?”

Outcomes are the effects or changes brought about by the curriculum or activities provided by an institution or program. In the process of specifying outcomes, a program/institution typically establishes measurable criteria for each outcome.

What are “outcomes assessments?”

Outcomes assessments answer the question "Did we, in fact, accomplish what we intended for our students, institution, local community, or other stakeholders?" It is a formal process for comparing actual results to the established outcomes criteria. Changes implemented as a result of these analyses can be documented and outcomes can be measured again in a following cycle to verify improvement. Measurements of outcomes show the degree of an institution’s or program’s effectiveness.


3

Why Should We Assess Student Learning Outcomes?

• Accreditation– Institutional (General Education)– Program (Discipline-specific)

• Strategic planning• Renewed focus on institutional/program effectiveness and

continuous improvement– Trend analysis (longitudinal/cross-sectional studies)– Benchmarking– Comparison with other institutions/programs

• Measure instructional effectiveness– Curriculum alignment– Teaching methods

• Facilitate student feedback/counseling/advising• Performance funding/making effective resource allocation decisions• Marketing• Demonstrate performance to stakeholders demanding

accountability and continuous improvement


4

Why Use Standardized Outcomes Assessments?

• National or other broad comparison group• Longitudinal data• Diagnostic feedback• Test design

– Validity– Reliability– Fairness– Representativeness


5

• Voluntary System of Accountability– A voluntary initiative for four-year public colleges and universities– A collaborative effort of the American Association of State Colleges

and Universities (AASCU) and the National Association of State Universities and Land-Grant Colleges (NASULGC)

– Designed to improve public understanding of how public colleges and universities operate

• College Portrait– An attempt at providing consistent, comparable, and transparent

information on the characteristics of institutions and their students– Intended to be used by students, families, policy makers, campus

faculty and staff, the general public, and higher education stakeholders

More information available at www.voluntarysystem.org

What is the VSA?


6

• A common reporting template only five (5) pages in length– MS Excel workbook format uses visual basic macros to convert

data into worksheets, graphs, and tables

• Majority of the data elements selected are from currently available data sources with established definitions and reporting conventions (e.g. the Common Data Set)

• Data elements divided into three standardized sections– Consumer Information

– Student Experiences and Perceptions

– Student Learning Outcomes

More information available at www.voluntarysystem.org

The VSA College Portrait


7

• Costs of attendance• Degree offerings• Living arrangements• Student characteristics• Graduate rates• Transfer rates• Post-graduate plans

The VSA College PortraitConsumer Information Data Elements

What information would be most helpful to prospective students and their families in deciding which university best fits their educational wants and needs?


8

• Results reported within six specific constructs shown to be correlated with greater student learning and development– Group learning– Active learning– Experiences with diverse groups and ideas– Student satisfaction– Institution commitment to student learning and success– Student interaction with faculty and staff

The VSA College PortraitStudent Experiences and Perceptions Data Elements

A snapshot of student experiences and activities and their perceptions of the college or university


9

• Institution description of how they evaluate student learning– Links to institution-specific outcomes (e.g. professional

licensure)

• Results from one of three standardized instruments which measure broad cognitive skills– MAPP is one of these assessments– Results reported two ways

• Value added (learning gains between freshman and senior years)

• Actual average test scores for freshmen and seniors

The VSA College PortraitStudent Learning Outcomes Data Elements

An attempt at standardizing evidence of student learning


10

• Timeline– Institutions can begin collecting and reporting information any time

after February 15, 2008

– Initially, the College Portrait does not have to be fully populated

• Institutions can continue to add data as it is gathered

• This includes data on a standardized outcomes assessment

• Measurement of learning outcomes is considered a pilot– Many institutions had not previously administered standardized

outcomes assessments on a large scale

– Institutions may take up to four years to choose their assessment and begin reporting results

– Standardized testing vendors have been permitted these four years to arrive at the common method for reporting value added

The VSA College PortraitGetting The Facts Straight


11

Take into account the academic ability of the students using a control variable (such as ACT/SAT scores) in a regression algorithm

Y = p0 + p1*x - p2*x

• Requires data from large groups to develop the “formula”• “Levels the playing field” based on expectations• Permits all VSA-approved outcomes measures to report on a

common scale across institutions of varying levels of selectivity• Each testing organization will use the same method to calculate

and characterize value added

What is value added? The VSA Methodology

Learning gains between freshman and senior year

Not the actual regression algorithm


12

Well Above Expected

Above Expected

At Expected

Below Expected

Well Below Expected

NOTICE… The scale is Descriptive – NOT Numeric!!!

What is value added? The Common Value Added Scale


13

• Value added score is descriptive – not numeric– There is no one qualifying score that determines whether an

institution is above, at, or below expectations

– The VSA value added methodology measures expected growth specific to an institution – not across institutions

– Data from a large number of institutions will be used to create the regression algorithm

• Most of the VSA-approved tests currently cannot report value added in this way (VSA methodology)– Studies currently underway

– Test vendors committed to reporting value added using the same methodology in the same timeframe

– MAPP users will be able to use data from current administrations to determine value added retroactively

• Retain the ACT/SAT scores of each MAPP cohort tested



14

• Study design– Cross-sectional (different cohorts of students)

• Faster, simpler, less costly• Freshmen and seniors• May be tested in the same year

• Frequency– At the conclusion of four year pilot, institutions will report

standardized test results at least once every three years• ACT/SAT scores may not be used as a proxy for freshmen

testing– VSA feels that value added cannot be measured accurately using

two different instruments- High correlations between admissions test and outcomes test do

not mean that they are measuring the same thing in the same manner



15

• Sampling– Recommendations from testing organizations are minimums

– Minimums are for reporting institution-level scores (value added)

– Oversampling required to obtain scores for specific subgroups

– Random stratified sampling recommended to ensure appropriate representation

– ETS experts will consult with your institution on the sample size and methodology best suited to your needs



16

The VSA College PortraitReporting MAPP on the College Portrait

Is that all there is?


17

“New kid on the block?”

Measure of Academic Proficiency and Progress (MAPP™)

20 year history as Academic Profile(jointly created and operated by ETS and College Board)


18

• Single, integrated measure of general education skills

• Focuses on academic skills rather than knowledge• Provides invaluable data for…

– Accreditation– Strategic planning– Curriculum improvement– Performance-based funding– Benchmarking– Determining “value added” or learning gains– Voluntary System of Accountability



19


Available in two administration lengths - Standard and Abbreviated

Standard Form for more individual student data

Abbreviated Form for the shortest available assessment of this kind

Multiple forms for pre-/post-testing Eliminates concerns about exposure or “test-retest effect”

Norm-referenced and criterion-referenced scores

Results can be used for benchmarking, determining “value added,” and curriculum improvement

Paper-pencil or Internet-based delivery Choices for any institution regardless of extent of technical infrastructure

Proctored or unproctored versions Unproctored versions are perfect for distance learning students and facilitating sampling of off-campus students

Features Benefits


20


All forms and versions equated Results from Standard/Abbreviated, Pre-/Post, Paper-Pencil/Online, and Proctored/Unproctored can be compared or combined

Customizable demographic questions Allows analysis of subgroups of importance to each institution

Ability to add multiple-choice questions Allows further examination of content, skills, or attitudes not covered by the core test

Comprehensive and customizable comparative data

Allows comparisons using a reference group of interest to each institution

Optional essay using e-Rater™ Use constructive-response module to gauge students’ writing progress

Features Benefits


21

MAPP Assesses Academic Skills

• College-Level Reading 25%• Critical Thinking 25%• College-Level Writing 25%• College-Level Mathematics 25%


22

MAPP Assesses Academic Skills

• Humanities– Selections of poetry, fiction, and nonfiction prose– Discussions of painting, music, or philosophical issues

• Social Sciences– History, Economics, Political Science, Psychology,

Anthropology, or Sociology

• Natural Sciences– Biology, Chemistry, or Physics

In The Context Of…


23

Two Versions of MAPPTo Meet Different Needs

Standard Form

• Covers all skills in one test• 108 multiple-choice questions• Administered in 120 minutes*• Generates all scaled scores and

proficiency classifications for both individuals and groups

• Provides demographic/ subgroup breakdowns

*Can be administered over two class periods

Abbreviated Form

• Covers all skills in one test• 36 multiple-choice questions• Administered in 40 minutes• Generates all scores for groups

but only total scores reported for individual students**

• Provides demographic/ subgroup breakdowns

**Additional data can be downloaded for institutional research and analysis but without certain student identifiers


24

Two Versions of MAPPTo Meet Different Needs

Standard Form Abbreviated Form

Standard Form A

108 questions

27 questionsper skill area

2 hours

AbbreviatedForm A1

36 questions


40 minutes

AbbreviatedForm A2

36 questions


40 minutes

AbbreviatedForm A3

36 questions


40 minutes

Delivered in matrix administration mode(or “spiraled”)

So few questions in each skill area = no skill subscores for individual students (reliability)

Indicates that Abbreviated Forms are equated back to the Standard Form. Notice that Abbreviated Forms are not equated to each other.


25

What Information Can We Get From MAPP?

• Norm-Referenced Scaled Scores– “Norm-referenced” means “a score that takes its meaning from

comparison against the scores of others– Such scores can be used easily for benchmarking or for determining

“value added” or learning gains by conducting a longitudinal or cross-sectional study

• Eight (8) Norm-Referenced Scaled scores reported– Total Score– Skills Subscores (Critical Thinking, Reading, Writing, and

Mathematics)– Context Subscores (Humanities, Social Sciences, and Natural

Sciences)

MAPP Score Types


26

What Information Can We Get From MAPP?

• Criterion-Referenced Proficiency Classifications– “Criterion-referenced” means “a score that takes its meaning from

comparisons against specific learning objectives such as a set of skills or abilities or specific content from a content domain”

– Such scores can be used easily for curriculum improvement

• Nine (9) Criterion-Referenced Proficiency Classifications reported– Mathematics (Level 1, Level 2, Level 3)

– Writing (Level 1, Level 2, Level 3)

– Reading (Level 1, Level 2)

– Critical Thinking

MAPP Score Types


27

MAPP Scaled Score Report


28

MAPP Comparative Data


29

MAPP Comparative DataReported by…

• Class Level/Credit Hours• Carnegie Classification

Visit the MAPP website at www.ets.org/mapp to view comparative data.


30

MAPP Proficiency Classifications Report


31

MAPP Proficiency Level DefinitionsProficiency levels are defined by a specific set of skills, abilities, or competencies mapped to each level in the design of the test.

Reading/Critical Thinking

To be considered Proficient at level 1 a student should be able to…- Recognize factual material explicitly presented in a reading passage - Understand the meaning of particular words or phrases in the context of a reading passage

To be considered Proficient at level 2 a student should be able to…- Synthesize material from different sections of a passage- Recognize valid inferences derived from material in the passage - Identify accurate summaries of a passage or of significant sections of the passage - Understand and interpret figurative language - Discern the main idea, purpose, or focus of a passage or a significant portion of the passage

To be considered Proficient at level 3 a student should be able to…- Evaluate competing causal explanations- Evaluate hypotheses for consistency with known facts- Determine the relevance of information for evaluating an argument or conclusion - Determine whether an artistic interpretation is supported by evidence contained in a work - Recognize the salient features or themes in a work of art - Evaluate the appropriateness of procedures for investigating a question of causation - Evaluate data for consistency with known facts, hypotheses or methods- Recognize flaws and inconsistencies in an argument


32

Standard MAPP Score Reports

• Summary of Scaled Scores (Total/Skills/Context)

• Roster of Scaled Scores

• Summary of Proficiency Classifications (numeric and graphical)

• Roster of Proficiency Classifications

• Scaled Score Distributions (Total/Skills/Context) (graphical)

• Individual Student Reports

• Subgroup Reports

• Demographic Reports

• Additional Questions Report

• Data download– Useful for analyzing smaller groups of interest


33

Data Presentation is Half The BattleFrom Standard MAPP Score Report…


34

Data Presentation is Half The Battle…to MS Excel spreadsheet…

2005 2006 2007 2008

Mean Score - Freshmen 112.5 111.5 112 113.5Mean Score - Seniors 115.5 115.5 117 117.5Gains (+/-) 3 4 5 4


35

Data Presentation is Half The Battle…to easy-to-understand chart…

0

20

40

60

80

100

120

140

2005 2006 2007 2008

Mean Score -Freshmen

Mean Score -Seniors

Gains (+/-)

…in less than 1 minute.


36

MAPP Demographic Reports

Fourteen Demographic Categories

• Class Level• Age• Gender• Ethnicity• Language Ability• Enrollment Status• Hours Worked• Credit Hours

Completed

• Employment status

• Overall GPA

• Program

• Intended/current major

• Core requirements completed

• Hours transferred


37

Sample MAPP Demographic Reports

MAPP

MAPP

Subgroup sample size must be large enough to provide meaningful data


38

Other MAPP Reporting Features

• All reporting via the web – no paper score reports• Reports downloadable as MS Word and MS Excel files• Ability to combine scores…

– …across cohorts– …across multiple administrations– …across paper-pencil and online– …across proctored and unproctored

• Data downloadable for analysis or import

Future enhancements…

• Custom Comparative Data report– Allows institution to select reference group from MAPP user list

• Item Information Report– Analysis of group performance on each question by skill area


39

MAPP is customizable with Institution-Specific Questions

• Additional Demographics (Subgroups)– Create up to 72 subgroups that the user defines

• Facilitates analysis of subgroups of importance to each institution

• Nine (9) demographic questions with up to eight (8) possible student responses

• Locally-Written Questions– Author and include up to 50 additional multiple-choice

questions• Allows further examination of content, skills, or attitudes

not covered by the core test

• Reported as percent correct for each response choice

• Individual responses to each locally-written question also reported in the data download


40

• Generalizing to the entire institution population– Sample mean vs. true population mean (Can you say anything

meaningful or unequivocal about 4,000 seniors based on a sample of 200 seniors?)

– There is no “magic number”

• Analysis of subgroups– Gender, ethnicity, major, etc.– Minimum = 50 students per subgroup* (# Subgroups X 50)

*For the abbreviated form. (There are no minimums for the Standard Form.)

– Ideal = based on statistical power

• Curriculum improvement– Estimates of proficiency classifications become more accurate with

larger groups

Sample should be arrived at consultatively and should be based on institutions needs and statistical properties like “power” and SEMean

Sampling Revisited When you need more than the minimum


41

• More skills assessed than just critical thinking and written communications– Reading– Math

• Maximizes data collection while minimizing testing time– Improves student motivation and willingness to participate– Reduces effects of testing fatigue

• Integrated skills design produces fair, accurate, and reliable results– All students sampled must be assessed in all skill areas

Choosing MAPP for VSA Beyond “Value Added”


42

• More uses for data than just value added– Curriculum improvement

• Criterion-referenced scores make analyzing curriculum easier than ever

– Benchmarking• Compare to peer institutions in a meaningful way

– …by Carnegie classification

– …by customized peer reference group

• Ability to customize– Addition of demographic subgroup questions of local

interest– Addition of up to 50 skills/content questions



43

• Ease and flexibility of administration– MAPP offers both paper-pencil and online versions– MAPP offers unproctored versions

• Sample distance learning students with no inconvenience to you or students

– Online versions offer immediate score reporting– Scores from all versions can be combined for aggregate

reporting– MAPP allows continuous on demand testing

• 365 days a year• 24 hours a day, 7 days a week with unproctored testing



44

• Administer MAPP to cohorts of 200 freshmen and 200 seniors– ETS scores the tests

– Scores available online

• Request the MAPP Learning Gains Report– Institution identifies cohorts to be included

– ETS provides institution with a spreadsheet for providing SAT/ACT scores of MAPP test takers

• Complete the spreadsheet with SAT/ACT scores• Receive the report within four weeks

MAPP users will be able to use data from current administrations to determine value added retroactively

• Retain the ACT/SAT scores of each MAPP cohort tested

Using MAPP for the VSAPractical Matters


45

• FAQs – Using MAPP for the VSA• Sample MAPP Learning Gains Report• Sample SAT/ACT Scores Spreadsheet• MAPP Learning Gains Report Request Form• White Paper

Measuring Learning Outcomes In Higher Education Using the Measure of Academic Proficiency and Progress (MAPP)

www.ets.org/mappinfo

Using MAPP for the VSAResources


46

Creating Assessment Instruments -Important Considerations

• Reflect the mission and values of the institution

• Link assessment data to improvement of student learning

• Measure explicit learning goals and objectives

• Gather data that is meaningful and useful


47

Choosing Assessment Instruments -Important Considerations

• What is the purpose of this test?• What are the intended uses of the test? • Who are the users of this test?• What aspects of the department/

program/institution do we want to measure?• What scores will be reported?


48

Questions and Comments?

• Visit the ETS Higher Education Assessment web pages at www.ets.org/mapp to learn more about the MAPP test

• Contact a MAPP expert toll free at 1-800-745-0269

• If you wish to participate in the MAPP value added study, contact Bill Wynne at 609-683-2006 or [email protected]

• Voluntary System of Accountability website• www.voluntarysystem.org• Overview of VSA• Overview of College Portrait and its data elements

http://www.voluntarysystem.org/

Documents

MAPP and VSA The Role of the Measure of Academic Proficiency and Progress in the Voluntary System of Accountability Bill Wynne Solutions Implementation