Assessment in service learning: Student learning outcomes

Julie A. Hatcher, Exec. Dir. Assoc. Prof. Philanthropic Studiesjhatcher@iupui.edu Kristin Norris, ABDnorriske@iupui.eduAssessment Director

Center for Civic & Social ResponsibilityUniversity of KansasMay 21, 2014

Outcomes for work/play shop• Basic intro to assessment (of x, y, z)• Implications of a “public work” approach at KU• IUPUI “case study”– CMG– NSSE

• Identify 1-2 AAC&U learning outcomes for either – Course– Program– Campus

• Identify next steps you will take on your assessment matrix

Current Support for “Civic”• Membership Organizations

– AAC&U– AASC&U– American Academy of Arts and Sciences– Campus Compact– Imagining America– NASPA

• Foundations– Bonner Foundation– Carnegie Foundation: Elective Classification for C.E– Kettering Foundation– Lang Foundation – Lilly Endowment– Lumina Foundation– Teagle Foundation– John Templeton Foundation

• Government– State Commissions of Higher Education– Massachusetts Board of Higher Education

Rigor in Research (Assessment)

• Rigorous thinking• Hypothesis exploration• Systematic approach• Builds upon prior research• Contributes to the field• Triangulation across inquiry• Information synthesis• Creates useful, relevant knowledge

(Patton, 2012)

Schoen’s Reflective Practice• Trait of a good professional• Reflection-in-action• Attentive to new knowledge• Test, adjust, reframe their models of

practice• Revise plans

Kolb’s Experiential Learning Cycle

Reflective Observation

Concrete Experience

Abstract Conceptualization

Active Experimentation

Reflective SL Practitioner

Practice

SL Course/Program/ Curriculum Design

Implications for you/othersProgram/Course Redesign

Theory

Research/SoTL

Formative AssessmentReflection ActivitiesClassroom Assessment

My Reflective Observations• Wine or peanut M&Ms• Important to identify a “north star”• Takes a significant amount of time• Year of Assessment in 1997 ish

• Highly connected to scholarship• We find our way by walking…• 3 semesters until “in the saddle”• Refine, improve reflection strategies• Ongoing improvement of processes• Tweaks

Boyte’s concept of Public Work• Places public work (not deliberation or

social justice) at the center of democracy• Common pool resources (Ostrom School)• Collective action – “we” not “they”• Lay participation is essential• Trait of a democratic professional (Dzur)• Conflict is intrinsic and healthy aspect• Co-generators of knowledge -- partners

Assessment as Public Work• Portland State case study -- resources– Assessing service-learning and civic

engagement: Principles and techniques• AAC&U – VALUE rubric case study– 16 rubrics; all available free– No need to reinvent the wheel

• IUPUI case study

Current Context at KU• Reflect – pair & share – wagon wheel• What are the key “pressures” or levers

that are motivating you personally to be here today?

• What do you hope to do, improve, advance as a result of being here today?

• Who do you need to bring to the table so this is done collectively?

Terms of Scholarship -- data• Assessment (What occurred?)– Formative – during the course/program

– Summative – end of the course/program

• Program Evaluation (What aspect?)– Program Evaluation Research

• Research (Why? What conditions?)– Qualitative– Quantitative– Mixed-methods

DefinitionService learning is a course-based, credit-bearing educational experience* in which studentsa) participate in an organized service activity that meets identified community needs*, and b) reflect on the service activity in such a way as to gain– further understanding of course content, – a broader appreciation of the discipline, and – an enhanced sense of personal values* and civic

responsibility.(Bringle & Hatcher, 1995)

Formative Assessment in SL• Classroom Assessment Techniques

(Angelo & Cross, 1995)– Exit Cards–Minute Papers–Mid-semester evaluation

• Reflection activities– Pair and Share– Journals - various types (DEAL model)– Value of informal/unstructured

• Community partner feedback– Coffee……and conversation

Summative Assessment -- Data• Course-based– Grades– End-of-Course survey; pre/post– Products

• Curriculum-based– E-portfolio–Digital stories– Exit survey for majors

• University-based– Counting; classifying– Longitudinal impact on students– Evidence of community impact

Assessment in SL Course• Are grades sufficient? (perhaps)• What literature review could help to “ground”

the assessment? (ex. Nutrition, Architecture)• How can you use/analyze reflection activities

as a source of data to document student learning?

• What pre/post learning could you assess?• What other products could you gather?• Multiple sources of data -- triangulation

Designing and Assessing SLClass/Program

Learning Objectives of Course - Curriculum - Campus

Selection of Community Partner and Service

Experience

Structured Reflection Activities Framed by Learning Objectives

Products are Created

Assessment of Student Learning and Products

Scholarship of Teaching/Learning SOTL

Service Learning Outcomes• Academic Development– Persistence and retention– Achievement and aspirations

• Life Skills– Racial tolerance– Cultural understanding

• Civic Responsibility– Commitment to community– Aspirations to volunteer

(Sax & Astin, 1997; UCLA/HERI)

(www.compact.org/resource/aag.pdf)

Meta-Analyses of SL• Requires a body of research• Psychology, business, communications

• Academic• Personal • Social• Citizenship

Case Study from IUPUI• Institutional culture and commitment• Assessment & Civic Engagement• Assessment Institute (Oct 21-23., 2014)

• Ethos in our work• Bob Bringle’s leadership• Scholarship and research

• Signature Center designation• IUPUI Book Series – Research on SL Vol 1, Vol 2• Boyer Scholars Faculty Learning Community• IUPUI Research Academy (each May)

Types of Citizenship Across Disciplines • Civic professionalism• Social responsibility• Social justice• Connected knowing: Ethic of care• Public leadership• Public intellectual• Engaged/public scholarship

(Battistoni, 2002)

Civic-Minded Graduates• “to what end? ”• Civic-Minded Professionals• John Dewey• Democracy and Education

• William Sullivan• Habits of the Heart• Work and Integrity

• Ernest Boyer• “public good” argument

CMG as a “North Star”A civic-minded graduate is one who a) is formally educated and b) has the capacity and orientation to

work with others c) in a democratic wayd) to improve the community.

(Hatcher, 2011)

Civic-Mindedness

“a person’s inclination or disposition to be knowledgeable of and involved in the community, and to have a commitment to act upon a sense of responsibility as a member of that community”

(Steinberg, Hatcher, Bringle, 2011)

Civic-Minded Graduate Model

Civic and Workforce Development

• Developing civic-minded graduates and professionals

• Developing workforce “soft” skills• Recognizing the importance of place

(Battistoni & Longo, 2012)

Domains of CMG: Civic Knowledge

•More than purely academic knowledge (dates, places, important civic or political events)• Knowledge of volunteer opportunities (ways to

contribute to society and of nonprofit organizations)• Knowledge of contemporary social issues (current

events and the complexity of issues in modern society)

(Steinberg, Bringle, & Hatcher 2011)

Domains of CMG: Civic Skills

• Communication and Listening (ability to communicate with others and listen to divergent points of view)• Diversity (understanding the importance of, and the

ability to work with others from diverse backgrounds)• Consensus-building (ability to work across difference to

come to an agreement or solve a problem)

Domains of CMG: Civic Dispositions

– Valuing community engagement (understanding the importance of service to others, and being actively involved in the community)

– Self-efficacy (have the desire to take personal action, with a realistic view that the action will produce the desired result)

– Social trustee of knowledge (feeling a sense of responsibility and commitment to use the knowledge gained in college to serve others)

CMG Survey• 30 Likert-type items (student self-report)• Knowledge, skills, dispositions, behaviors• 6-point response format

• Demographic items• Student activity items• Paper or online administration• Adaptable: “My education at IUPUI” – course, or

major, depends on the research question

Assessment Tools• Course Evaluation• SL End –of-Course Survey• Domains to include for your campus?

• Student Learning Outcomes• Civic-Minded Graduate Scale• Civic-Minded Graduate Narrative• Civic-Minded Graduate Rubric• AAC&U VALUE Rubric (Civic Engagement)

Implications of CMG--CMP• Program Design• Alternative Break; days of service

• Service-scholarship Applications• Plater Medallion• Consultations with faculty and departments• New work in faculty development and

graduate education

Pair and Share• Put on your “disciplinary hat”• Why is it important to you as a…landscape

architect, student affairs professional…to include civic learning in your course/program?

• What type of civic learning do you hope occurs for your students?

• Who are your stakeholders, and why would they care about these outcomes?

Learning Objectives• List 2 civic learning objectives in your

community engaged program/course.

• (How do these align with broader curriculum?)• (How do these align with accountability for

external program review, if applicable?)• (How do these align with the campus mission

for student learning?)

“Deep Learning”• High Impact Practices (e.g., “HIPs”)• Distinct value of service learning• Additive value of multiple experiences• See AAC&U, Ashley Finley, Director of Research and

Assessment

• NSSE Data (12 items)• Higher-Order learning• Integrative learning• Reflective learning

Participation in Service Learning Courses

The independent variable, participation in serving learning courses, was derived from NSSE survey question 1K:

In your experience at your institution during the current school year, about how often have you done each of the following?

k. Participated in a community-based project (e.g., service learning) as part of a regular course

Participation in Service Learning Courses

   IUPUI

 

Urban13

 Public

Research

 NSSE

Sample

Freshmen 56% 38% 38% 41%

Seniors 58% 40% 43% 48%

Deep Learning

The dependent variable deep learning was comprised of three different scales. Reliability analysis was conducted for higher-order learning (α=.83), integrative learning (α=.73), and reflective learning (α=.83).

The data file was then split into freshman and senior students so the analysis could be conducted on these two populations separately.

Higher Order Learning Questions (α=.83)

During the current school year, how much has your coursework emphasized the following mental activities?

Applying theories or concepts to practical problems or in new situationsAnalyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its componentsMaking judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusionsSynthesizing and organizing ideas, information, or experiences into new, morecomplex interpretations and relationships

Integrative Learning Questions (α=.73)

In your experience at your institution during the current school year, about how often have you done each of the following?Worked on a paper or project that required integrating ideas or information from various sourcesIncluded diverse perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignmentsPut together ideas or concepts from different courses when completing assignments or during class discussionsDiscussed ideas from your readings or classes with faculty members outside of classDiscussed ideas from your readings or classes with others outside of class (students, faculty members, co-workers, etc.)

Reflective Learning Questions (α=.83)

During the current school year, about how often have you done each of the following?

Examined the strengths and weaknesses of your own views on a topic or issue

Tried to better understand someone else’s views by imagining how an issue looks from his or her perspective

Learned something that changed the way you understand an issue or concept

The Findings

An independent-samples t-test evaluated differences in reported deep learning skills between students who participated in one or more service learning courses and those students who did not participate in service learning courses.

Deep learning skills of higher-order learning, integrative learning, and reflective learning were all higher for both seniors and freshman who participated in service learning course(s).

IUPUI Freshman

*p<.05, 2-tailed significance

Construct

  

# of Items

  

Mean (Overall) N=524

 Mean

(Service Learning) N=305,

58%

 Mean(No

Service Learning) N=219,

42%

Mean Difference

(SL and No SL)

Reliability Effect Size Sig.

Higher Order Learning

4  3.05 3.09 2.99 .10 .83 .08 .085

Integrative Learning 5 2.62 2.75 2.43 .32 .73 .27 .000*

Reflective Learning 3 2.72 2.82 2.58 .24 .82 .16 .000*

IUPUI Seniors

*p<.05, 2-tailed significance

Construct

  

# of Items

  

Mean (Overall) N=998

 Mean

(Service Learning) N=588,

59%

 Mean(No

Service Learning) N=410,

41%

Mean Difference

(SL and No SL)

Reliability Effect Size Sig.

Higher Order Learning

4  3.23 3.36 3.03 .33 .86 .24 .000*

Integrative Learning 5 2.81 2.99 2.57 .42 .72 .34 .000*

Reflective Learning 3 2.86 2.96 2.72 .24 .83 .16 .000*

Implications

Results contribute evidence of student learning at the institution level

Findings are consistent with prior research on participation in service learning and improved student outcome measures (Astin et. al., 2000)

Provide a rationale for institutions to support faculty who engage with the community partners to develop service learning courses

Matrix FrameworkCore Concept Key Indicator of

learningMethods to gather data

Source of the data

AAC&U Learning Outcome 1

AAC&U Learning Outcome 2