Kenneth Janz, Ph.D.AVP and CIOWinona State University

Improving Assessment Efforts Integrating Educational TechnologyWednesday, March 16, 2011, 8:30 – 9:15, Seville West EDUCAUSE Midwest 2011, Chicago, Illinois

Ken Graetz, Ph.D.Director, TLT

Winona State University

Overview of Presentation• Overview of Winona State University (WSU)• e-Warrior: Digital Life and Learning Program• Assessment of Educational Technology• e-Warrior Assessment Plan• Key Findings• Ongoing Assessment Challenges• Questions

Founded in 1858 (Normal School)Public State AssistedWinona, Minnesota

9,000 students

Required Participation Dual-platform lease program since 1997$480 semester

e-Warrior Video Goes Here

Winona is Unique……..• Space Utilization (Currently at 93%)

– Anytime, Anywhere Learning– Computer Labs

• Computer Self-Efficacy• Access to Software• Innovative Pedagogy

– e-books– enhanced communications

• Faculty Can Count On What Students Can Bring To Class

Interesting Data……

The federal IPED’s report indicates Winona State’s five-year undergraduate graduation rate leads all other MnSCU Universities by at least 10 percent (IPED, 2009). 1997 WSU graduation rate was just over 40%.

Attracting Students

Faculty Recruitment

Rankings

Awards

*Core based on Fink (2003, p. 60 – 101)

Better wireless access

Shifting pedagogies

Device backlash

New devices Greater academic integration

Budget cuts

Aging laptop programs

More applications

What Makes the Target Hard to Hit?

Where Are We Going• Winona State

University Technology Master Plan

• Runs July 2010 – June 2013

• Framework for integrating and prioritizing technology related needs

*Core based on Fink (2003, p. 60 – 101)

Research Questions Based on Conceptual Model

Measuring How We Are Doing

Building Longitudinal Metrics

• Student Surveys• Faculty Surveys• Alumni Surveys• Faculty / Student Focus Groups• Collect Course Information• Presentation / Publications• Existing Student Success Data• Facilitating Activities• Unit HLC Assessment

Multiple Measures (1 of 2)• Annual Faculty and Student Technology Survey• Faculty and Student Training Counts• Faculty and Student Training Evaluation Survey• Alumni Survey• Faculty Portfolio • Archival Log Data • Help Desk Satisfaction Survey• Faculty and Student Focus Groups

Multiple Measures (2 of 2)• Review Selected Course Information• Review Selected Course Assignments • e-Warrior Digital Life and Learning Program meeting

attendance and minutes• Showcase Survey • Presentation and Publication Counts • Number of inquiries and visits to learn about the

program.• Existing Student Success Data • Educause Core Data• Application and Course Specific Measures

Key Findings and Measures

Student-Related Questions• Are students satisfied with the tools?

– Do first year students find the program “lame”– Do Seniors become disillusioned with the program?– Are students adding tools not provided through the program?

• Do students use the tools for academic purposes extensively?– Do they use the tools more for academic vs entertainment?– Do they use the tools inside the classroom?– Does academic tool use vary by year in school?

• How do students evaluate their instructors’ tool use?– Do perceptions vary by year in school?

Student satisfaction (5-point scale)Item Mean

I use my laptop frequently for class related purposes. 4.38

The use of a laptop is important to my academic success. 4.37

I am satisfied with the software provided in the program. 4.10

The laptop provided is of good quality. 4.01

I was adequately trained to use my laptop. 3.93

The program policies are clear. 3.93

What grade would you give the e-Warrior Digital Life and Learning Program overall?

• “A” = 38% overall• “B” = 53% overall• Both are satisfied, but Seniors grade

the program lower

F

D

C

B

A

0 20 40 60 80 100

Total

First Year

Seniors

On average, approximately how many hours per day do you use your laptop?

• Normally distributed, averaging 5-6 hrs/day

• No differences between first years and seniors

Less than 1 hr/day

1-2 hrs/day

3-4 hrs/day

5-6 hrs/day

7-8 hrs/day

9-12 hrs/day

More than 12 hrs/day

0 20 40 60 80 100

TotalFirst YearSeniors

Of the total average time that you use your laptop during a typical day, how much of the time do you use it for class-related purposes?

• Positively skewed, averaging 3-4 hrs/day

• No differences between first years and seniors

• Use their laptops fornonacademic purposesabout 2 hrs/day

Less than 1 hr/day

1-2 hrs/day

3-4 hrs/day

5-6 hrs/day

7-8 hrs/day

9-12 hrs/day

More than 12 hrs/day

0 20 40 60 80 100

Total

First Year

Seniors

On average, in the courses you use your laptop while in class, how frequently do you use it in class?

• Students use laptopsin class 2-3 coursesper term

• Students do notuse laptops inclass everyday

• Seniors use laptops in classmore freq. thanfirst year

Never

Sometimes, but not regularly

About half the time the course meets

Most of the time the course meets

Every time the course meets

0 20 40 60 80 100

Total

First Year

Seniors

Application frequency (5-point scale)Item Mean

Access course materials 4.68

Communicate with your instructors 4.06

Collaborate with other students in your classes 4.02

Create class-related presentations 3.97

Take a quiz or exam 3.84

Take notes 3.51

Added Software• Communication Tools

– Skype– Yahoo Messenger

• Creativity/Productivity Tools– FinalCut Express– Audacity– Google Chrome– Picasa– Evernote– Soundflower

• Entertainment Tools– VLC Player

• Academic Tools– LaTeX– Dim Sum– Google Earth– Nook Study, other ebook readers

Use D2L

• I feel that all instructors should use the D2L system as it is a great way for students to know where they stand in the course, because they are able to view their grades regularly. Therefore, it would be nice if instructors would update grades for students as well on the site. Another reason D2L is a great system is that it helps with contacting classmates, and is great for instructors to update with all of the information from each class.

• I like D2L but a lot of the Minne professors don't seem to use it much. Too much paper still handed out!! not green

More Engagement

• Using photos or YouTube clips to supplement their lesson is one thing that a couple of teachers have done, that I found very beneficial.

• Instructors need to understand that just reading off slides does not teach students effectively. Instead, incorporate examples or class activities or even more hands on learning. If it is a proven fact students learn more from hand on activities, you think the college would require material to be taught through hands on techniques.

• Don't just read right off the slide you made, it's boring and no one learns anything.

Less Technology in the Classroom?

• Thank you finally a question I have a strong opinion on. If your going to require laptops and charge as much as you do for them, no professor should be allowed to say you can't use them to take notes in class. I am well aware of the distraction they present. Figure something out, you cant be a laptop school if you have to take notes in one or two classes each semester by notebook.

• I, personally, would prefer the instructors to use less technology because it is easier for me to learn by doing something myself, rather than a computer/calculator to do it for me.

• Positive correlation between cumulative GPA and frequency of laptop use in class; moderated by year in school– First year: r = +.19– Senior: r = +.23

Relationship With GPA

Student-Related Questions• Are students satisfied with the tools?

– Do first year students find the program “lame”– Do Seniors become disillusioned with the program?– Are students adding tools not provided through the program?

• Do students use the tools for academic purposes extensively?– Do they use the tools more for academic vs entertainment?– Do they use the tools inside the classroom?– Does academic tool use vary by year in school?

• How do students evaluate their instructors’ tool use?– Do perceptions vary by year in school?

• Are faculty satisfied with tools and services?

• Are faculty using tools in their courses effectively?

Faculty-Related Questions

Item Mean

Technology Support (i.e., contacted TLT with a specific technical question about a particular tool)

4.54

Technology Consultation (i.e., contacted TLT for general advice on the application of technology)

4.68

Faculty and Staff Workshops (i.e., attended a live session in person or online) 4.36

Customized Student Training (e.g., requested development of student workshops or online training)

4.71

Online Learning Materials (e.g., accessed movies or other online help provided through TLT)

4.50

Project Support (i.e., worked with TLT staff on a specific, technology-related academic project)

4.64

Blog, Wiki, and Moodle Services (i.e., used a blog, wiki, or Moodle course on TLT servers)

5.00

Special Application Services (i.e., developed/used a custom or special application on TLT servers)

4.80

Item Mean

D2L Course Merge/Copy Service (i.e., requested D2L course sections be combined or copied)

4.37

Multimedia Lab Services (e.g., used TLT video editing and production services) 4.67

Video and Audio Streaming Support (i.e., requested assistance with streaming recorded events)

4.20

Special Event Planning and Support (e.g., conference, festival) 5.00

ITV Technical Support (e.g., TLT assisted scheduling or connecting an ITV meeting) 4.77

Class//Meeting Room Hotline (i.e., requested immediate technical assistance in a campus room)

4.66

Multimedia Equipment Checkout (e.g., borrowed camera, projector, headset, clickers) 4.71

Faculty Exchange

Faculty Exchange Video Goes Here

Ongoing Assessment Challenges• IMS Analytics • Assessment Related to Faculty

Development

Questions