An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment

Methods

Direct Methods

Conventional or Technology-Enabled Standardized Instruments: Tests and Inventories

Description: Historically, standardized instruments, such as objective tests, have served

as the primary direct method to assess student learning. Content or disciplinary experts

identify the standard content, knowledge and tasks that students should know and be able

to perform. In addition, they determine what constitutes levels of achievement based on

the construction, design, and sequencing of questions or prompts. Students’ achievement

is referenced against other student groups’ achievement on the same instrument, referred

to as norm-referencing. Primarily designed to make decisions about students,

standardized instruments perform a gate keeping role. They certify competence in a

profession or field, such as in the case of licensure examinations for nursing students, or

program-level knowledge or skills, such as in the case of general education tests. Results

are also used to place student in appropriate courses or to identify level of achievement at

points in students’ studies, such as in the case of rising junior examinations.

What They Provide:

- Content and tasks developed by external experts within fields or

1

programs of study

-Psychometric approach to assessment that values quantitative methods of

interpreting student achievement

-Evidence of what students know or can do within the universe and framework of

questions, prompts, and tasks of an instrument

- Evidence to make gate-keeping decisions such as professional certification or

end-of-study achievement or to meet state mandates that value

measurement as proof of learning

-Evidence to track student learning if instrument can be used formatively and if

results have utility for programs and the institution

-Quick and easy adoption and efficient objective scoring

-History of validity and reliability studies

-One possible source of evidence within an institutional commitment to assessing

student learning through multiple lenses

What Standardized Instruments Do Not Provide:

2

-Evidence of the strategies, processes, and ways of knowing, understanding, and

behaving that students draw upon or apply to represent learning

-Evidence of the complex and diverse ways in which humans construct and

generate meaning

-Alignment with institution- and program-level learning outcome statements and

students’ learning histories

-Realistic timeframes or contexts that reflect how humans solve problems, seek

additional information or resources, correct mistakes, or reposition their

thinking. Students respond within a timeframe that might well

affect their decisions or actions, such as deciding to make a last-minute guess

among

options in a question.

-Highly useful results that directly relate to pedagogy and educational practices.

Results relate to the construct of the instrument itself and what it is designed to

measure. Patterns of student performance reported in scales or scores

identify discrete areas of performance, such as a skill level, that identify

strengths and weaknesses in curricular or co-curricular attention.

However, these patterns do not assist in learning about why students responded in

3

the ways they did. Or did they learn successful strategies for selecting or

making “good guesses?”

Some Examples:

Instruments that test general education knowledge and abilities include the following:

* Academic Profile: http://www.ets.org/hea/acpro

* Collegiate Assessment of Academic Proficiency (CAAP):

http://www.act.org/caap/index.html

* Collegiate Learning Assessment (CLA)

* MAAP

Instruments dedicated to measuring specific skills include:

* ACCUPLACER : (http://www.collegeboard.com/highered/apr/accu/accu.html

* Watson-Glazer Critical Thinking Appraisal

* California Critical Thinking Skills Test

* Tasks in Critical Thinking

* Reflective Judgment Inventory

* Measure of Intellectual Development

* e-Write, a component of ACT’s COMPASS/ESL system

Examples of achievement tests in a particular field of study or profession include:

* Graduate Record Examinations' Subject Tests:

4

http://www.gre.org/pbstest.html#testreg

* The PRAXIS Series: Professional Assessment for Beginning Teachers:

http://www.ets.org/praxis/index.html

* Area Concentration Achievement Tests: www.collegeoutcomes.com

* Graduate Management Admission Test: http://www.mba.com/mba

Examples of inventories that assess students’ knowledge include: include:

* Force Concept Inventory (FCI)--designed to assess the initial knowledge state

of students before they begin undergraduate physics courses. Students

respond

to a series of force-related problems or statements that reveal their knowledge

(Halloun and Hestenes,1985).

Knowledge surveys

Locally Designed Tests and Inventories

In a collective and shared commitment to assessing student learning, core working groups

may well determine that no appropriate standardized instruments exists that aligns with

institution- and program-level outcomes. That decision generates the design of local tests

or inventories or use of existing instruments that have an institutional history of providing

useful and reliable results. Technological advancements, such as WebCT

5

(http://www.webct.com/transform) and Blackboard

(http://www.blackboard.com/products/ls/index.htm) offer an online environment for

constructing some locally developed instruments.

What Local Tests or Inventories Provide:

-Strong alignment of content and format with learning outcome statements and

course-based assessment methods students’ have experienced along their

learning histories

-Useful results that can be interpreted within the local contexts of teaching and

learning and then used to improve student learning

-Opportunity to establish local instrument criteria that reflect what an institution

and its programs value in educational practices.

-Opportunity for faculty, staff, administrators, students, teaching assistants,

tutors, intern advisors, advisory board members, and institutional

researchers, for example, to contribute their perspectives on what should be

assessed and how it should be assessed.

What Local Tests or Inventories Do Not Provide

-Immediate reliability and validity results that verify content, construct, format

and consistency in scoring, unless instruments have been pilot tested and

evaluated over several semesters. Thus, time to pilot test an

instrument for an institution’s representative student populations is a

necessary component of an institutional commitment to assessing for learning.

6

Conventional or Technology-enabled Authentic Performance-based Methods

Authentic performance-based methods prompt students to represent their learning in

response to assignments and projects that are embedded into their educational

experiences. These methods value divergent thinking and responding, as opposed to

convergent thinking, most typically represented in standardized tests. Focusing on how

students think, problem solve, react, interpret, or express themselves becomes the focus

of these kinds of direct methods. Further, these methods can easily be embedded into

students’ continuum of learning, providing evidence of their growth over time often

demonstrated in self-reflective writing and responses to feedback from those who

contribute to their education, such as peers, internship advisors, external reviewers or

evaluators, faculty and staff.

What They Do Provide:

-representation of integrated learning

-direct alignment with students’ learning experiences

-opportunities for students to reflect on and receive formative feedback about

their learning and development

-student-generated opportunities to demonstrate learning, as opposed to test-

generated occasions

_opportunities for students to create as in multi-media contexts

What They Do Not Provide:

-Easily quantifiable evidence given the complexity they capture

-Efficient scoring opportunities

7

Range of Authentic Assessments:

Portfolio or e-portfolio—collection of multiple kinds of student-generated

texts stored electronically or in paper form. Developments in technology now

make it possible for students to create digital portfolios. This method of

storage and collection provides a longitudinal representation of learning,

demonstratinghow students make meaning within their contexts for learning

through assignments, projects, narrative self-analyses and self-reflection.

They provide an opportunity for students to demonstrate how they integrate

learning over time from multiple learning experiences and opportunities

within the curriculum,co-curriculum, and learning experiences that extend

beyond their formal educational experiences. They provide a valuable source

of evidence for institution- and program-level learning by providing a range of

texts that represent student learning. Webfolios, digital learning records, or

electronic learning records are other current terms for this method. The

development of 2.0 electronic portfolios offers students a context to

demonstrate their learning processes as well as learning products. Two major

national resources that focus on technological developments in electronic

portfolios are the following: 1. the newly founded global academic

organization, Association for Authentic, Experiential and Evidence-based

Learning (AAEEBL: http://www.aaeebl.org/) and 2. the community of

practice, Electronic Portfolio Action and Communication (EPAC:

http://epac.pbworks.com). Cambridge, Cambridge, and Yancey’s 2009

8

publication, 2.0 Electronic Portfolios: Emergent Reseearchon Implementation

and Impact (Stylus Publishing,LLC), provides a taxonomy of electronic

portfolios and describes the potential of 2.0 electronic portfolios  For

information about Alverno College's Diagnostic Digital Portfolio, go to:

http://www.ddp.alverno.edu/.

Learning Record Online

A running record of students’ learning is the Learning Record Online (LRO).

Originally developed for K-12 and now being developed for higher education,

the LRO provides formative and summative evidence of students’ learning

within their curricula and against agreed upon criteria and standards of

judgment. Like a portfolio, the LRO provides ongoing evidence of students’

emerging learning within contexts for learning, including evidence of their

reflective process, their metacognition. Facilitated by online technology,

educators and others interested in learning about students’ development are

able to aggregate or disaggregate groups to draw inferences and make decisions

about students’ progress and eventual levels of achievement: .

http://www.cwrl.utexas.edu/~syverson/olr/

* Capstone project —a culminating independent research, collaborative, or professional

project at the end of students’ careers that provide evidence of how students solve

representative higher order disciplinary, professional or interdisciplinary problems often

represented in more than one kind of text, that is in writing as well as in speaking or in

visual texts, such as poster presentations. These projects provide evidence of how well

9

students integrate, synthesize, and transfer learning; in addition they also can provide

evidence of how well students integrate institution-level outcomes. This method can also

be integrated as a formative means of assessment. The beginning of a second year of

graduate study or the beginning of a third year of students’ undergraduate study might

index a time for students to demonstrate accumulated learning. Senior theses or senior

research projects are also examples of capstone projects that provide opportunity for

observers to assess students’ masterly level within a field of study, discipline, or

profession.

* Performances, products, creations—required over time as well as at the end of

students’ studies, their work represents how they interpret, express, and construct

meaning. Traditionally, the arts’ faculty have formatively assessed students’

performances to provide them with immediate feedback that shapes their future

performances. This approach provides not only students with immediate feedback but

also faculty and others who contribute to students’ learning.

* Visual Representation--Representing learning visually through charting, graphing,

mapping, for example, provides students with alternative ways to represent their

learning, often a practice within disciplines such as mathematics and the sciences.

Mathematicians often represent developments in their thinking in mind maps:

http://jonathan.mueller.faculty.noctrl.edu/toolbox/examples/seaver/

mindmappingtask.htm

10

Visual representation offers a way to assess how well students make connections or

understand a concept. In addition, visual representation extends students’ repertoire of

making meaning, developing a versatility in forms of representation that respond to the

different needs of audiences, contexts, and the purposes of communication. The Biology

Teaching Homepage provides examples of different kinds of conceptual maps, well as

procedures for incorporating them into students' learning.

http://www.fed.cuhk.edu.hk/~johnson/misconceptions/concept_map/cmapguid.html . The

Innovative Learning Group presents different forms of thinking maps.

file://A:\ILG%20%20Thinking%20Maps%20Information.htm

* Case studies-- used over time, as well as at the end of students’ studies, case studies,

often used in business programs, provide opportunity to assess students’ problem-

solving abilities within a major program of study or along students’ continuum to

determine how well they are integrating the learning expressed in institutional learning

outcomes—knowledge, perspectives, abilities, values, attitudes. Parallel case studies

used over time provide evidence of students’ abilities to solve representative

disciplinary, professional, or more generalized problems. In addition, they provide

evidence of student’s writing. Adding to the dimensionality of traditional paper- or

document-based case studies are online case studies that incorporate multi-media into

the case study and hyperlinks for users to read or see related resources. Harvard’s

* Professional or disciplinary practices—engaging students in practices that prepare

them for the kinds of problems, activities, or situations individuals address not only in

11

their fields of study but also as contributors to society and local communities. The Harrell

Professional Development Center in the University of Florida’s College of Medicine has

created an environment that permits observation of medical students interacting with

patients. Audio visual equipment captures, records, and displays these interactions.

Replaying these interactions provides opportunities for faculty and students to assess

students’ knowledge, understanding, behaviors and dispositions. file://A:\Harrell

%20Professional%20Development%20and%20Assessment%20Center.ht

*Team-based or Collaborative Projects--With rare exception, humans work with other

humans during most of their lives in a range of workplace, social, and community

environments. Team-based or collaborative projects are direct methods that enable

assessment of individuals’ knowledge, understanding, behaviors and attitudes, as well as

their ability to work with others to achieve a final product or solve a problem. Often

videotaped, groups of faculty, staff, and students themselves have access to immediate

results that inform students as well as educators. Alverno College’s institutional example

on page x illustrates assessment that focuses on individual students as well as their

collective achievement. Community service projects provide another source of evidence

of how well students apply or transfer or integrate learning to solve a problem as well as

an opportunity for students to self-reflect on how they contributed and what and how they

learned from the experience.

Wikis and podcasts represent two technologically-based assessment opportunities:

student teams create new work and contributions are date and time stamped to document

12

who contributed material to a project. For example, at the end of a program of study

teams of students might collaborate to create a culminating assignment on a wiki site,

designed to assess how well majors integrate knowledge, abilites, habits of mind, ways

of thinking and knowing.

Educational online gaming-

Internships or Practica—How students actually apply or transfer their cumulative

learning can be assessed in internship or practica. That is, authentic experiences become

a direct method of assessment, providing opportunity for students to demonstrate the

dimensions of their learning within the context of a real environment. A step beyond

simulations, these direct methods assesses how well students translate their cumulative

learning into actual practice.

Oral examinations—often used at the end of a professional program or a graduate

program, as in an oral doctoral defense (University of Arizona, Graduate College:

http://grad.admin.arizona.edu/degreecert/ppfoedc.htm), oral examinations provide

opportunities for students to represent how well they integrate learning and apply it to

solving a case study or problem, responding to guided questions, or presenting a product.

Measurement Research Associates provides some guidelines for conducting effective oral

examinations: http://www.measurementresearch.com/media/standardizedoral.pdf.

13

Simulations—are now possible through virtual reality programs such as Second Life

perhaps the most used virtual reality Virtual labs, virtual patients Data mining

merlot or Ariadne offer studnets that opportunity

Conversion--converts one form of representation into another form of representation, such as into numerical or statistical, scientific, humanistic, sociological, psychological, or artistic forms of representation

Those that provide opportunity for students to construct their own interpretaions, or identity new or unepected problems

Data mining--collects, records, analyzes, and interprets data (Merlot)

Problem identification/solution—identifies a new or unique problem or poses a new or unique way to view or solve a problem (Here’s a problem and the solution; are there other solutions?)

“Conceptual collisions” (Federation of American Scientists)—brings “previously disconnected knowledge sources into contact for the first time.”

Projection: identifies consequences, results, resulting conditions, challenges, actions, responses

SecondLife sites such as Virtual Ability Island, the Theorist Project, and Sci-Lands

Conventional and Technologically-enabled Indirect Methods

Helpful in deepening interpretations of student learning are indirect methods, methods

that focus on perceptions of student learning by asking students or others to respond to a

set or series of questions. Indirect methods function to complement direct methods rather

than to substitute for them.

What Indirect Methods Provide:

14

-Evidence of students' attitudes, perceptions, experiences

-Evidence that may help to explain student performance levels

What Indirect Methods Cannot Provide:

-Work that represents evidence of student learning unless an instrument asks

students to produce a text as evidence

Some Representative Examples:

-Focus groups with representative students to probe a specific issue

that may have been identified in a survey or identified in patterns of

student performance as a result of formative or summative assessments.

Van Aken and collaborators describe the successful design of a focus group

to obtain perceptions of historically under-represented students enrolled in

an engineering program (1999).

-Interviews with groups of students representing the institutional population.

Tracking a cohort of students is one way of assessing learning for

formative and summative purposes.

-Student, alumni, employer and faculty-staff surveys and questionnaires

that provide information about students’ or others’ perceptions of students’

educational experiences and the institutions’ impact on their learning.

Alumni questionnaires and surveys provide a retrospective view of

15

graduates’ educational experience and create an opportunity for

them to recommend improvements in education based on what is relevant to

their current employment, profession, or graduate education. Faculty-

staff surveys or questionnaires provide perceptions of student learning—

that is what students are able and not able to demonstrate based on classroom-

based assessments or observations of student behavior.

-ACT's surveys for adult learners, alumni, entering students, withdrawing

students, and for institutional services: www.act.org

-College Student Experiences Questionnaire (CSEQ):

Community College Survey of Student Engagement (CCSSE)

http://www.ccsse.org/aboutccsse/aboutccsse.html

-Community College Student Experiences Questionnaire (CCSEQ):

http://www.people.memphis.edu/~coe_cshe/CCSEQ_main.htm

- The National Center for Higher Education Management Systems'

Comprehensive Alumni Assessment Survey: www.nchems.org

-National Survey of Student Engagement (NSSE):

http://www.iub.edu/~nsse/

16

-Noel-Levitz Student Satisfaction Inventories:

http://www.noellevitz.com/library/research/satisfaction.asp#ssi

--Incoming Freshman Assessments (such as CIRP):

--First-year expericne surveys such as YFCY:

-Small Group Instructional Diagnosis (SGID)—a facilitated small group

interview process conducted by a trained interviewer who asks students

to identify what and how they have learned and what kinds of learning

obstacles they are facing or have faced. Used initially at mid-point in a

course to provide feedback to a faculty member about how students are

or are not learning, this method can also provide useful formative

assessment data at the program-, department-, or institution-levels ,

providing students’ first-hand explanations for dominant patterns of

weakness in their work:

http://www.miracosta.edu/home/gfloren/sgid.htm

--Student Assessment of Their Learning Gains (SALG)—a customizable

online end-of-course survey that asks students to rate the components of

a course, such as lab work or discussion groups, and the degree to which

these components contributed to their learning. This survey could well be

used for program-, department-, or institution-level assessment if a sample

of students were asked to rate how well and why the components of

learning they have been explosed to have contributed to their learning of

17

stated learning outcomes in their program of study . Given that the survey

is customizable, institutions can add additional kinds of questions that

probe students’ perceptions of their learning.

http://www.miracosta.edu/home/gfloren/sgid.htm

Inventories that track how well students have changed or shifted attitudes, values,

sensitivities, perspectices based on self-perception or responses to situations or case

studies.

* Some Representative Examples include:

--Global Perspective Inventory (GPI)—46-item on-line inventory designed

to assess the effect of cultural experiences on learners’ global

perspectives through three dimensions: cognitive, intrapersonal and

interpersonal (https://gpi.central.edu/)

--Intercultural Development Inventory (IDI)-- 44-item inventory based

paper and pencil instrument designed to assess the extent of an

individual’s intercultural sensitivity along a continuum that ranges

from ethnocentrism to ethnorelativism, identifying a person’s ability to

shift from denial of difference to integration of difference

(http://www.intercultural.org).

* A list of current inventories frequently used in student services is available at:

The Office of Institutional Research and Planning, The University of Texas at

Arlington. Student Affairs-Related Outcomes Instruments: Summary

Information.

18

Inventories that gauge students’ , such as the Intercultural Development Inventory

(IDI) and the GPI that guage the extent to which studnets have shifted or changed

attitudes, values or even perspectives representa yet another kind of indirect methodof

Assessment that may provide evidence of how much or to what degree students change

their views based on their learning either in the curriculum or co=curriculum. Often used

as pre- and post-tests in a program or service, these kinds of assessments provide

evidence of how much or to what degree students chage their views, attitudes or beliefs

Examples of Graphic Organizers

http://www.eed.state.ak.us/tls/Frameworks/mathsci/ms5_2as1.htm#graphicorganizers

19