Assessing an Integrated Course, Content, Library and

ePortfolio SystemNick LaudatoBarbara FreyDan Wilson

Richard P. OravetzCenter for Instructional Development

& Distance Education

University of PittsburghCopyright © 2005 Nicholas C. Laudato, Barbara Frey, and the University of Pittsburgh. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the authors.

Agenda Background

University of Pittsburgh ePortfolio Committee Bb Content System Assessment Project

Approach Findings Observations Conclusions

Pittsburgh

GreensburgJohnstown

Bradford

Titusville

University of Pittsburgh

Founded in 1787Five campuses26 Academic CentersGraduate,

undergraduate, professional, and continuing education programs

30,030 Fall-term FTE 4,533 full- and part-time

faculty members

Blackboard at Pitt Pitt was an early

adopter (CourseInfo V 1.0)

Training required for a faculty control panel

Adopted by all schools and campuses

20-40% annual increase in adoption

Reaches 75% of student population

216

839

1,197

1,630

2,329

3,219

3,804

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

AY1999

AY2000

AY2001

AY2002

AY2003

AY2004

AY2005

Course Sections using Blackboard

Background: ePortfolio Committee

Initiated in December 2003 Goals:

Identify needs and interests Establish criteria for an e-portfolio

system Discuss policy and procedure issues and

make recommendations to the Provost Examine major software options to

assess their capabilities and potential viability

What are ePortfolios?

An electronic portfolio is a collection of multimedia objects and artifacts that provides a record of, and reflection on, activities and achievements, and can be accessed differentially for multiple purposes and reviewers

- University Committee on Electronic Portfolios

Motivators for the Project Recognized need to

Gain real experience with ePortfolios Generate real sample ePortfolios

Interest in extensions to Blackboard Improve the management of course

content Multiple sections of the same course Sharing materials across multiple instructors

Provide an eReserves capability integrated with Blackboard courses

The Blackboard Content System

Content Management - sharing and reusing files across sections, courses, and departments

ePortfolios - organize, document and share information for assessment, advising, career planning, and academic growth

eReserves - incorporate library resources directly into online course materials

Virtual hard drive - network file storage space easily accessible from any computer at any time

BbCS Assessment Project Goals

Evaluate the Blackboard Content System (BbCS) as a tool for Enabling faculty to manage their own

course content Supporting multiple sections of a course Supporting school, departmental, and

program libraries of course materials Supporting e-Reserves Metadata schemes and search

capabilities

… BbCS Assessment Project Goals

Evaluate ePortfolio for use by Students Staff/faculty for career and

professional development Assess the Bb Content System

Storage and processor requirements User support requirements

BbCS Assessment Project Approach

Participants Faculty Librarians Instructional Designers and Technologists Students

“Kick the Tires” Assessment Measures

Interviews and Surveys Student Group Project: RFP Evaluation

BbCS Assessment Project Timeline

Preliminary planning in March 2005 Planned to address summer 6-week sessions

Acquired access to the system on April 11, 2005 pitt.blackboard.com

Summer Session began May 9, 2005 Included 12-week sessions

Originally targeted to end midterm (June 30, 2005)

Extended through the end of the summer term (end of August)

Project Participants Courses

Management Information Systems, Laudato and DeSantis

Information Systems Analysis and Design, Zhou CNS Clinical Practicum I, Henker Retrieving Information, Knapp Individual Research I, Tomer and Alman

ePortfolios Students in Career Services and in MIS course Faculty and staff within CIDDE

Findings1. Managing Course Content2. Support for Multiple Sections3. Libraries of Course Materials4. eReserves5. Metadata and Search6. Student ePortfolios7. Professional Development Portfolios8. Support Requirements

1: Managing Course Content

Conceptual Picture of Content SystemMyContent

Metadata

Adding Content to Course Sections

Metadata

MyContent Course Section

Course Section

Personal, Course, Institutional,

and Library ContentMyContent

InstitutionContentLibraryContent

Course SectionCourseContent

What Users Said … Library and Information Science

Faculty “Tested WebDAV on both PCs and

Macs with positive results.” “Faculty will have to work smarter to

reap the benefits of the content system. An object oriented repository will have a direct impact on changing the way we do business.”

What Users Said … Nursing Faculty

“With WebDAV, I can get to my content from anywhere in the world.”

“Nursing courses build on one another, so being able to see what previous instructors have taught is valuable.”

Information Science Faculty “Using the Content System is not

worth the effort if you are only teaching a course one time.”

What Students Said … “The content system is easy to access …” “… good way to gather and manage

scattered data in order to create academic, personal, career documents.”

“Most people are familiar with the Windows environment as well as web browsers. Being web-based, Blackboard is an excellent choice to provide these things.”

“I believe this is a revolution …”

What Students Said …

VS S N D VD NA

6. the WYSIWYG editor. 0% 0

10%

1

30% 3

30%

3

30%

3

0% 0

7. the file storage features of the Blackboard Content System.

30%

3

60%

6

10% 1

0% 0

0% 0

0% 0

8. WebDAV. 10%

1

50%

5

30% 3

0% 0

0% 0

10%

1

9. the metadata component of the Blackboard Content System.

10%

1

40%

4

50% 5

0% 0

0% 0

0% 0

Please rate your overall satisfaction with ...

Simplified Model of a Bb Course

Bb Course with Content System

MyContent

CourseContent

InstitutionContent

LibraryContent

Metadata

Course Side Content Side

Course Archive?

Pre-App Pack 3 Interface to BbCS

Content System Content

Note: This is actually an App Pack 3 screen shot

Course Integration in App Pack 3

Course “Faculty Information” Area

2: Support for Multiple Sections

Without the Content System Managing two

sections of the same course requires two section maintenance efforts

Faculty merge rosters to avoid this work

With the Content System: No Change!

Course Side Content System Side

What is Content to the BbCS?

In Bb, a course folder or learning unit contains: Text Images Multimedia

Content Interactive Content Assignments Quizzes Surveys

What We NeedCourse Side

We must be able to embed a complete learning object (course folder or learning unit) into the content system

Content System Side

3: Libraries of Course Materials

Tested capabilities with mock school and departmental libraries and a sample library of images for use in teaching languages

Faculty speculated that “Core courses by different instructors can

share an area and offer students similar experiences and knowledge bases”

“It is important that courses build on the foundation of previously-created courses”

Unable to evaluate with real case

Institution Content

4: eReserves

eReserves for a Course Section

Library Content

eReserves

Course Section 1

Course Section 2

Course Section 3

Electronic Journal 1

Electronic Journal 2

Electronic Journal 3

5: Metadata and Search Bb provides four metadata schemas:

general, IMS, Dublin Core, and custom Uploaded an image collection and

populated it with custom, IMS, and Dublin Core values

Limited experience with this function High cost of creating the metadata

along with the single-use nature of this assessment project did not justify the effort

Custom Metadata

What Users Said … Information Science Faculty

“Cannot simultaneously search across multiple metadata schemes”

“Can neither import nor export metadata”

“Data entry is tedious and error-prone” Library and Information Science Faculty

“Metadata capabilities are good, but faculty will not use it unless it can be imported.”

6: Student ePortfolios Students in a graduate MIS class

Assigned two portfolios and directed to use the content system

Competency file Job search tool

Assigned a group project to evaluate the Bb Content System against ePortfolio functional and technical requirements

Student workers in Career Service office

The Blackboard ePortfolio

The Portfolio Creation Wizard

Adding Content to ePortfolios

MyContent

Resume

Competency File

What Students Said About ePortfolios…

“… it is like we have a goal and direction we can follow.”

“… would help incoming students save and organize academic documents ... it seems like it will be able to deliver these things.”

“… allows for a cohesive view of academic achievement that can be maintained throughout the academic career and beyond.”

What Students Said …

VS S N D VD NA

1. the e-Portfolio component of the Blackboard Content System.

0%0

90%

9

10% 1

0% 0

0% 0

0% 0

2. the e-Portfolio creation wizard.

20%

2

50%

5

10% 1

10%

1

0% 0

10% 1

3. the ability to download an e-Portfolio.

30%

3

40%

4

30% 3

0% 0

0% 0

0% 0

4. the ability to share an e-Portfolio with internal users.

40%

4

60%

6

0% 0

0% 0

0% 0

0% 0

5. the ability to share an e-Portfolio with external users.

20%

2

60%

6

10% 1

0% 0

0% 0

10% 1

Please rate your overall satisfaction with ...

The MIS Group Assignment Two independent groups assigned a

project to assess the ePortfolio “… each group will be asked to assess

the ePortfolio component of the Blackboard Content System.” 

Given the requirements portion of the final report of the ePortfolio Committee 59 functional and technical

requirements

Degree the Requirements Were Met

Requirement CategoryRequirement Achievement

Percentage

Access 100%

Other features 100%

Review and Feedback 91.7%

Content 89.3%

Usability 87.5%

Reporting and Output 86.1%

Technical Requirements 79.2%

Training and Support 71.4%

Overall Achievement 86.4%

Results of Requirements Analysis

Critically Important

4

 42 12, 56 7, 18, 43, 51, 55, 57, 58

1, 8, 9, 19, 20, 21, 22, 23, 25, 26, 28, 29, 30, 31, 34, 39, 40, 48, 53, 54

Moderately Important

3 17, 43 11, 36, 44 16, 50, 52 2, 3, 5, 6, 10, 13, 14,

15, 24, 27, 32, 35, 37, 38, 41, 45, 46, 47

Somewhat Important

2  4 59 33

Not Important at

All1

 

1 2 3 4

Not Achieved

At All

Somewhat Achieved

Moderately Achieved

Fully Achieved

ePortfolio Templates

Copy Portfolio Building Block

7: Professional Development Portfolios

Nursing Faculty Member “I love the ePortfolio for globally

collaborating with colleagues to share articles, files and clinical websites.”

Instructional Designer “Users take responsibility for building

the matrix as a professional development tool that demonstrates mastery of a list of criteria. The WYSIWYG editor needs work, but the matrix concept is valuable.”

App Pack 3 Enabled Matrices

8: Support Requirements Adopted some Bb tutorials and created

Camtasia recordings For Course

Content system complexity mandates some faculty training

For ePortfolios MIS students received a 10 minute

orientation Career services students received an

hour orientation and telephone support

… Support Requirements Processor and memory requirements

Unable to measure Storage requirements

Default quotas much too low for faculty Increasing defaults in the system does

not impact existing users (get it right up front)

Count on at least 100 MB per account, with some up to 1 GB

Plan for ongoing training and support

Observations The content system must be even

further integrated with the course side Must be able to embed course objects,

such as folders containing quizzes, into the content system

App Pack 3 upgrade was a move in the right direction

Course archive must address content areas

WYSIWYG editor is not WYSIWYG

… Observations Student response to ePortfolio

exceeded our expectations Had not considered it “best of breed”

Bb templates are NOT portfolio templates but rather are Web page templates Our building block allowed us to create

ePortfolio template functionality ePortfolios need additional controls and

capabilities to handle reflections and institutional components

… Observations Metadata functionality needs to be

improved Easier more effective data entry Import and export functions Ability to simultaneously search across

metadata schemes (against related attributes)

What Students Said …

SA A N D SD NA

1. An e-Portfolio system should be available on an ongoing basis to support your academic work.

40% 4

50% 5

0% 0

10% 1

0% 0

0% 0

2. The e-Portfolio component of Blackboard should be available on an ongoing basis to support your academic work.

30% 3

50% 5

20% 2

0% 0

0% 0

0% 0

3. The Blackboard Content System should be available on an ongoing basis to support your academic work.

50% 5

30% 3

20% 2

0% 0

0% 0

0% 0

4. Creating an e-Portfolio is a valuable academic activity.

40% 4

50% 5

10% 1

0% 0

0% 0

0% 0

5. The e-Portfolio component of Blackboard contributed to your learning.

30% 3

50% 5

20% 2

0% 0

0% 0

0% 0

Please rate your level of agreement with the following statements.

Conclusions The Blackboard content system is still

new and evolving, but worth the effort The assessment team is

recommending that the University acquire the content system

Remember to fill out the online session

evaluation!

Barbara Frey

Questions?

Nick Laudatowww.pitt.edu/~ciddeweb

Speaker 1 Speaker 2