Enhancing OSP for Programmatic and Institutional Assessment Lynn Ward and John Gosney Indiana...

Enhancing OSP for Programmatic and

Institutional Assessment

Lynn Ward and John Gosney

Indiana University

IUPUI: Institutional Profile

• Founded 1969 with a strong local mission• Blended campus• Metropolitan research university• 20+ schools (15 with professional/pre-

professional foci)• Commuter campus, ~30,000 students

(~20,000 undergraduates)

Approach to General Education

• Early 1990s - General Education based in the schools—distributive model

• Accreditation prompts internal reflection• Campus mandate for change, specifically a

centrally coordinated approach and specific learning outcomes for general education

• 1998 – Campus adopts a competency or ability-based model

3

Principles of Undergraduate Learning (PULs)

• Core Skills – written and oral communication

– ability to comprehend, interpret, and analyze texts

– analytical (quantitative reasoning)

– Information and technological literacy

• Critical Thinking• Integration and Application of Knowledge• Intellectual Depth, Breadth, and Adaptiveness• Understanding Society and Culture• Values and Ethics

Assessment Needs• Document and demonstrate the effectiveness of

IUPUI’s approach to general education (HLCNCA and ICHE)

• Document student achievement in programs subject to specialized accreditation (Education, Engineering, Visual Communications, etc.)

• Standard reports that aggregate and summarize assessment data across courses and programs

• Filter and group on demographic and academic criteria

5

Original Vision

6

Challenges• Site-centric nature of OSP tools; no way to easily

aggregate data across sites

• No tools to simplify management of very large sites

• Customization also makes it difficult to aggregate data; each department uses a different evaluation form and rating scale.

• No canned reports; every report requires an experienced XML programmer understands underlying data structures

• Academic programs more concerned with their own disciplinary outcomes than PULs

7

Current Vision: Phase 1: Goal/Outcome Linking

• Instructor or program administrator creates and publishes goal set; goal set becomes aggregation point

• Instructors in program can link any course assignment, matrix cell, or wizard page to one or multiple goals

• Students can attach examples of their work directly to one or multiple goals.

• Standardization of evaluation form elements makes it possible to aggregate and report data across courses and programs

Goal/Outcome Linking

EDUC 301

EDUC 401 PracticumPortfolio

Principles of Teacher Education

Filtering and Grouping

Secondary Education

Phase 2: Goal/Outcome MappingProgram

OutcomesProgram

Outcomes

Institutional Outcomes

Institutional Outcomes

Campus-level Aggregation via Mapping

IUPUI

Publishing and Linking Items to Matrices

13

Create and Publish Matrix

14

Manage Site Associations

15

Associate Site(s) with Matrix

16

Create Assignment in “Associated” Site

17

Link Assignment to Matrix Cell(s)

18

Student View of Linked Assignment

19

Linked Assignment in Matrix Cell

20

Opened Assignment in Cell

21

Linkable Tools

• Assignments

• Matrices (cells can be linked to other cells)

• Wizards (pages can be linked to cells)

22

Other OSPEnhancements

The Problem

Matrix authors must select forms and evaluators in each cell, even when the same choices are used in every cell.

24

The Solution: Matrix Defaults

25

Use All Defaults

26

Override All Defaults

27

The Problem

• No workflow support for reviewers (providers of formative feedback)

• Students cannot solicit feedback from peers, advisors, etc.

• No way for individuals who are not CIG members to provide feedback

28

Reviewer Workflow

• Submit for Review button

• Email Notification• Eventually …

– Recipient notification– Visual indicator of new

feedback

29

Student Initiated Feedback

30

External Reviewers

• Email notification provides direct link to cell (Sakai UserID and Password is required)

• Eventually …– Reviewer dashboard in My Workspace to

aggregate pending feedback requests

31

The Problem

Submitted artifacts are forever frozen in My Workspace >Resources, which precludes users from reorganizing files and folders at a later time.

Also, students do not understand why some files in their personal resources cannot be moved or deleted.

32

Solution: Attachment Helper

33

Direct upload from local storage device.Direct upload from local storage device.

The Problem

Users in evaluator (or reviewer) role can open all cells in a users matrix, even if not selected as an evaluator for the cell.

More granular access control is needed to support range of implementation scenarios from highly sensitive and secure to open and collaborative.

34

The Solution

• Cells can only opened by the designated evals/reviewers

• Revision of matrix permissions to provide much greater flexibility and granularity in progress

35

The Problem

Cells that have been evaluated and returned for additional evidence or other modifications look just like cells that have never been submitted.

36

The Solution: Returned Status

37

What’s Next?

38

For Fall 2008

• Standardized evaluation form and reports • Enhanced evaluator workflow and dashboard• Reviewer dashboard• Auto-population of portfolio sites based on

membership of associated course sites• Linking to wizard pages, and overall wizard

improvement

39

Questions?

Lynn Ward, leward@iupui.edu

John Gosney, jgosney@iupui.edu

40