Dr. Stephen Landry, CIODr. Eric Fountain, IT Assessment CoordinatorDanielle Mirliss, Instructional Designer

Using External Data Sources to Increase Confidence in Assessment Conclusions

Who We Are …

Seton Hall University

Mid sized, Catholic affiliated university in suburban NJ, 15 miles from New York City.

4,400 Undergraduates, 50% residential.

Two campuses: Main Campus in South Orange, NJ and Law School in Newark, NJ.

Approx. 350 Full Time Faculty on the South Orange Campus.

Research Intensive (Research II) Institution.

Institutional Change @ SHUSHU made a major commitment to institutional change

in its 1996 Strategic Plan

Seton Hall became a ubiquitous computing campus, providing laptop computers to the learning community as part of tuition and fees

Seton Hall increased enrollment while increasing academic requirements for admission

Seton Hall added new programs and faculty

Seton Hall increased emphasis on assessment and accountability

Ubiquitous Computing @ SHU

Seton Hall University’s Mobile Computing Program is designed to ensure all members of the learning community have access to learning resources

which are increasingly online.

Program has three interdependent components:

Access: The University licenses students the use of a laptop as part of their tuition and fees;

Curricular Integration: The University provides support and incentives to faculty to use technology in T&L;

Services and Support: Network (Wired and Wireless), Help Desk, Laptop Repairs, Insurance, Training, etc.

Implementing Ubiquitous Computing1994 – 1996: Development of SHU Strategic Plan

1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School

1997: Mobile Computing Pilot Project expands to 300 incoming Freshman

1998: Mobile Computing Program launched for all incoming Freshman

2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented

2002: SHU graduates first mobile class

2004: Tablet PC Pilot Project

Mobile Computing and Curriculum Development

Seton Hall recognized that access to technology is necessary for the changes envisioned in the learning environment, but not a

sufficient condition.

Teaching, Learning and Technology Center

Provides support and training for faculty in their use of technology in their teaching

Internal Grant Programs:

Curriculum Development Initiative

Faculty Innovation Grants

Implementation of Classroom and Network Technology

Teaching, Learning, and Technology Roundtable

Mobile Computingand Assessment

In 1997 Seton Hall recognized it had a unique assessment opportunity

Mobile Computing Assessment Program has two objectives:

To assess the impact of ubiquitous computing on the learning environment

To provide prompt feedback to the Mobile Computing Program planning team to address potential implementation issues with the program

Assessment Methodology and Resources

Foundations of the Assessment Movement

Research Traditions Contributing to the Assessment Movement:

• Student Learning in College

• Retention and Student Behavior

• Evaluation and Scientific Management

• Mastery Learning

A B

CD

A. Reflect

B. Plan

C.Act

D.Observe

Assessment Process Cycle

SkillDemonstration

Performance Evaluations

Tracking Systems

Faculty Fellows

Surveys

ePortfolios

Focus Groups

Types of Assessment

National Studies

Studies Focused on Students:

• National Survey of Student Engagement (http://www.iub.edu/~nsse/)

• Higher Education Research Institute (http://www.gseis.ucla.edu/heri/heri.html

•CIRP Freshman Survey: profile of entering freshman class. •CSS College Student Survey: follow up of CIRP cohorts. •YFCY Your First College Year

• Flashlight (http://www.tltgroup.org/programs/flashlight.html)

Studies Focused on Faculty:

• Faculty Survey of Student Engagement (http://www.indiana.edu/~nsse/fsse/index.htm)

National Studies

Studies Focused on Technology:

• EDUCAUSE Core Data Survey (http://www.educause.edu/apps/coredata/index.asp?bhcp=1)

• Campus Computing Project (http://www.campuscomputing.net/)

• Cost of Supporting Technology Services (http://www.costsproject.org/)

Assessment in the Teaching, Learning and Technology Center

Current Initiatives

• Mobile Computing Assessment Team

• Project Assessment

• TLTR-Assessment Committee

Mobile Computing Assessment Team

Cross-Campus, Cross-Discipline• Provost’s Office• Institutional Research• Faculty• TLTC

Multiple Projects• Mobile Computing Assessment Survey• Extant Data Analysis• Large-Course Redesign• Classroom Study• Multi-Campus

Mobile Computing Survey

R1. What is the effect of the Mobile Computing Program on student satisfaction?

R2. What is the effect of the Mobile Computing Program on technology use?

R3. What is the effect of the Mobile Computing Program on desired educational outcomes?

Research Questions

Results

•Attracted to Seton Hall University by the availability of technology at the University and the infusion of technology in the curriculum, as suggested by student reported positive influences on the decision to attend the University,

• Satisfied with the Mobile Computing Program as well as with the laptop and support services once enrolled at the University,

• Making good use of the technology available to all students, at least in terms of certain types of technology use, and.

• Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment.

The typical Seton Hall University student, whether representing all respondents, or particular groups of students (groups identified by gender, race/ethnicity, and residence status), is generally:

Web-based Online Courses

Mixed mode Hybrid Courses

Enhanced Courses with electronic resources and exercises

Course Level Assessment Strategies

• Collaborative effort between Provost’s Office, faculty, and TLTC

• Student and Faculty focus groups

• Development and implementation of pilot survey

Hybrid Course Assessment

Tablet PC Highlights

MCAT – Future Plans

Expand Faculty Participation

Expand Membership to Include Student Affairs and Student Government

Organize and Publicize Results

Approach Question of Classroom Use

FlashLight

Comparison Studies

Using External Data Sources to Increase Confidence in Assessment Conclusions

An Example from Seton Hall University using results from the 2003 National Survey of Student Engagement

What NSSE Measures

Student exposure to and participation in educational practices that promote engagement and student

learningInteraction with faculty in & out of the classroom

Active & collaborative learningHigh expectations and academic challenge

Enriching educational experiences

Student perception of a supportive campus environment helping them to succeed

NSSE Questions - Active & Collaborative Learning

NSSE 2003 - Active & Collaborative LearningSeton Hall Freshmen vs Peers & National

45.9

40.141.8

30

40

50

60

SHU Doctoral-Intensive

National

NSSE Questions - Faculty Student Interaction

NSSE 2003 – Faculty Student Interaction Seton Hall Freshmen vs Peers & National

42

35.337.2

30

40

50

60

SHU Doctoral-Intensive

National

NSSE Technology QsTechnology Use

SHU Students SHU Carnegie Peers

Time online for academic - 6+ hours / week 55% 2003 SHU FR 40% Doct-Intensives FR 51% 2003 SHU SR 44% Doct-Intensives SR

Time online for any reason - 6+ hours / week88% 2003 SHU FR 77% Doct-Intensives FR81% 2003 SHU SR 72% Doct-Intensives SR

NSSE Technology Qs

Technology Use & Active LearningSHU Students SHU Carnegie Peers

How often required to use IT - % Very Often59% 2003 SHU FR 49% Doct-Intensives FR61% 2003 SHU SR 56% Doct-Intensives SR

Used T to make presentations - % Very Often47% 2003 SHU FR 34% Doct-Intensives FR56% 2003 SHU SR 53% Doct-Intensives SR

NSSE Technology Qs

Technology Use & Faculty ContactSHU Students SHU Carnegie Peers

Email ask instructor to clarify assign - % Very Often51% 2003 SHU FR 30% Doct-Intensives FR57% 2003 SHU SR 39% Doct-Intensives SR

Email to express ideas to instructor - % Very Often22% 2003 SHU FR 13% Doct-Intensives FR28% 2003 SHU SR 16% Doct-Intensives SR

NSSE Technology Qs

Technology Use & Active/Collaborative LearningSHU Students SHU Carnegie Peers

Work in teams in class using IT - % Very Often27 % 2003 SHU FR 14 % Doct-Intensives FR25 % 2003 SHU SR 19 % Doct-Intensives SR

Communicate with classmates on-line to complete academic work - % Very Often

51% 2003 SHU FR 27 % Doct-Intensives FR49 % 2003 SHU SR 39 % Doct-Intensives SR

NSSE Technology QsTechnology Use & Active Learning / Reflection

SHU Students SHU Carnegie Peers

Gain new insights into course material from on-line discussions - % Quite a bit+

43% 2003 SHU FR 37% Doct-Intensives FR31% 2003 SHU SR 32% Doct-Intensives SR

Express ideas to instructor by email not comfortable saying in class - % Very Often

22% 2003 SHU FR 13% Doct-Intensives FR28% 2003 SHU SR 16% Doct-Intensives SR

Conclusions

Questions?

Contacts:Dr. Stephen Landry, CIO – landryst@shu.edu

Dr. Eric Fountain, IT Assessment – fountaer@shu.edu

Danielle Mirliss, Instructional Designer – mirlisda@shu.edu

ResourcesInternet Resources for Higher Education Outcomes Assessment - http://www2.acs.ncsu.edu/UPA/assmt/resource.htm#nsse

AAHE Assessment Forum - http://www.aahe.org/assessment/assess_links.htm

Assessment in Higher Education - http://ahe.cqu.edu.au/