The new e-assessment?

The new e-assessment? The new e assessment? Leveraging online platforms for competency-based outcomes assessment

Jennifer Hill, Ed.D.Matt Serra, Ph.D.Matt Serra, Ph.D.

Office of Assessment, Trinity CollegeDuke University

assessment.aas.duke.edu

A copy of this PowerPoint is available at bit.ly / AACU_2014_Hill

1

The annual meeting’s RFP asks ga probative question:

What are the implications of a hifti h i shifting emphasis

from credit hours to competencies?to competencies?

2

The annual meeting’s RFP asks

Put another way:

H d i iti g

a probative question:

How do universities, purveyors of liberal education, negotiate

What are the implications of a hifti h i

educa o , ego a e the growing societal focus on students’

t f t shifting emphasis from credit hours to competencies?

mastery of concrete skills? What does this mean for our missions, to competencies? mean for our missions, pedagogies, and learning outcomes?

3

An example:

In 2013 the U.S. Department of Education d i it ’ l t ll approved a university’s proposal to allow use

of federal financial aid to students seeking degrees based on demonstrated degrees based on demonstrated competencies, rather than term-based, multi-credit coursework.Parry, M. (2013, April 18). Competency-based education advances with U.S. approval of program. Chronicle of Higher Education. http://chronicle.com/blogs/wiredcampus/u-s-education-department-gives-a-boost-to-competency-based-education/43439

4

Another example:

The University of Wisconsin's “extension system ill t t t b d l i will start a competency-based learning

program, called the Flexible Option, in which students with professional experience and students with professional experience and training in certain skills might be able to test out of whole courses on their way to getting a degree.”Carlson, S. (2013, September 30). Competency-based education goes mainstream in Wisconsin. Chronicle of Higher Education. http://chronicle.com/article/Competency-Based-Education/141871/

5

Quantitative reasoning

C iti l thi ki

Hypothetically, such a model could be

Critical thinking

Ethical reasoningstructured around key learning objectives such as:

Global perspectives and engagement

objectives such as:

And the individual student would advance through the

i l h h curriculum as he or she demonstrates mastery of specific learning outcomes.

6

Without debating the merits of these i ti th l t th tinnovations, we can nevertheless suggest that

this shift encourages this shift encourages brainstorming the metrics by which educators judge the which educators judge the quality of a competency-based curriculum.

7

Duke, in part, uses the credit-hour model of degree attainment (34 hours) to record student progress. ( ) p g

Trinity College students also complete their choice of courses in specific areas of knowledge and modes of courses in specific areas of knowledge and modes of inquiry to signify breadth and depth of intellectual engagement.

Cross-cult. Inquiry (2)

Ethical inquiry (2)

Foreign Language (1-3)

Research (2) Science, Tech., & Society (2)

Writing (3)

Arts, Lit., &P f Performance (2)

Civilizations (2)

Natural Natural Sciences (2)

QuantitativeStudies (2)

S i l Social Sciences (2)

8

How do we judge whether our graduates meet our expectations for meet our expectations for mastery in various domains of the liberal education?Cross-cult.

Inquiry (2)Ethical inquiry (2)

Foreign Language (1-3)

Research (2) Science, Tech., & Society (2)

Writing (3)

Arts, Lit., &P f

education?

Organizing student Performance (2)

Civilizations (2)

Natural

g glearning into competencies is a crucial conceptual framework for Natural

Sciences (2)

QuantitativeStudies (2)

S i l

conceptual framework for gen. ed. assessment.

Social Sciences (2)

9

We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning

i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g

[ and ]

Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and

l i l l b ti f l iexplain local observations of learning.

10

We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning

i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g

[ and ]

Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and

l i l l b ti f l i

How do we do this at

k ?explain local observations of learning. Duke?

11

Matriculate assessment initiative

Our purposes are

to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and

to follow the development of these competencies over time.

Project design allows (1) wide engagement among incoming students, (2) collection of data from three separate nationally-normed instruments, and (3) efficiencies of administration data analysis and reportingadministration, data analysis, and reporting.

12


What

To whom

When

How

13


What: Administration of several assessment instruments i t t d t ’ t i l ti t D k E h t t prior to students’ matriculation at Duke. Each targets a

general education student learning outcome.

To whom To whom

When

H How

14


What

To whom: Voluntary sample of incoming first year (non-transfer) students, A&S and Engineering

When

How

15


What

To whom

When: Project started in summer 2010 and continues jannually. Early communications start in April, with periodic reminders in July and August. Data collection ends at the start of first year student orientation.

How

16


What

To whom

When

How: Deploy three online, externally hosted instruments via a central project homepage. Data are returned to us b tby our partners.

17


18


19


Defining Issues Test (DIT-2)ethicaldevelopment.ua.edu

Reasoning about Current Issues Test (RCI)reflectivejudgment.org

Global Perspectives Inventory (GPI)gpi.central.edu

Critical thinking Assessment Test (CAT)tntech.edu/cat/home

20


Publicity: Blurb in “Blue Book”. Email from Director of Academic Advising Center Class Facebook page Academic Advising Center. Class Facebook page posting. Direct email from our office.

Incentives

Consent

Return rates

Challenges

Findings Findings

Next steps21


Publicity

Incentives: Drawing among eligible students. Awards and award values vary. “Hierarchy” of awards.

Consent

Return rates

Challenges

Findingsg

Next steps22


Publicity

Incentives

Consent: Consent statement is embedded on initial project homepage, and on each instrument’s website.

Return rates

Challenges

Findingsg

Next steps23


Publicity

Incentives

Consent

Return rates: Surprisingly high in year 1, fluctuating slightly over time.

Challenges

Findingsg

Next steps24


Publicity

Incentives

Consent

Return rates

Challenges: Monitoring completions on externally hosted b R t d i i t ti t i di t web apps. Remote administration prevents immediate

distribution of incentives. Maintaining high return rates.

Findings

Next steps

25


Findings:

Our voluntary samples of study participants are large and tend to be representative of their respective incoming classes classes.

Results at the pre-test tend to be stable across incoming classes. classes.

Individual students’ results and Duke’s overall results are emailed to them as early as possible. Content of the mailing is confidential, secure, contextualized, and written in accessible language.

26


Next steps:

Continue reviewing pre-test/baseline data for each matriculate class.

First study cohort graduates spring 2014; plans for postFirst study cohort graduates spring 2014; plans for post-assessment are underway.

When reviewing longitudinal data, we will be looking at iCollege results overall and breaking them down to explore

differences in learning gains by subgroup (e.g., major, co-curricular experiences).

Continue to explore other instruments that may be relevant for our learning objectives.

27

Documents

The new e-assessment?