OFFICE OF ASSESSMENT JULY 11, 2012 Non-Academic Assessment: Developing an Assessment Plan

OFFICE OF ASSESSMENTJULY 11, 2012

Non-Academic Assessment: Developing an Assessment

Plan

Goals of Workshop

Current assessment contextWhat’s new this yearIntroduce new Higher Learning Commission

CriteriaReview key items on the Assessment Plan

templateWalk through new online reporting form

2

Assessment Context3

Assessment Context

Assessment Context Last year participating in the Higher Learning

Commission’s (HLC) Academy for the Assessment of Student Learning

New HLC Criteria effective January 1, 2013 Campus focus on 2020 Strategic Plan alignment

Assessment Reporting Themes Focus on use of data Reflect on changes and impact on learning Streamline reporting process

Future Goals Longer reporting cycle Integration with annual reporting

4

What’s New This Year

Online reporting Fewer questions Questions “tagged” with New HLC Criteria Optional question on New HLC Criteria Report quality threshold to 70%

5

HLC New Criteria

Guiding Values, includes:1. Focus on student learning4. Culture of continuous improvement5. Evidence-based institutional learning and self-

presentation9. Mission-centered evaluation

Anatomy of Criteria Criterion—broad statement

Core Component—areas of particular focus Sub-Component—further elaboration on Core

Component, but not exhaustive (and not always included)

Some criteria now specifically mention certain units (e.g., advising)

6

HLC Criteria7

HLC Criterion One. Mission

The institution’s mission is clear and articulated publicly; it guides the institution’s operations. Core Component 1.A.

The institution’s mission is broadly understood within the institution and guides its operations. Subcomponent 2.

The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.

Criterion Title and

broad statement Core

Component statement—particular aspect of

criterion on which to

focusSub-component—specific area from

which evidence is expected

HLC New Criteria

The Five Criteria1. Mission2. Integrity: Ethical and Responsible Conduct3. Teaching and Learning: Quality, Resources, and Support4. Teaching and Learning: Evaluation and Improvement5. Resources, Planning, and Institutional Effectiveness

Assumed Practices Won’t check, but we are responsible to ensure/remediate Notable is B.2.c.1): Faculty participate substantially in:

analysis of data and appropriate action on assessment of student learning and program completion.

8

Upcoming Assessment Workshops

Developing an Assessment PlanThursday, July 19th, 10-11 am

Completing the Assessment Report Thursday, July 12th, 11-12 amTuesday, July 17th, 1-2 pm

Note: workshops/forms overlap

9

AN OPPORTUNITY TO STEP BACK AND PLAN EFFECTIVE ASSESSMENT AND ANALYSIS

Creating an Assessment Plan

10

Section 1: Current Unit Status

1. Explain how your unit supports 2020 Strategic Plan of YSU.

Removed mission; strategic plan operationalizes mission

11

HLC Criterion One. Mission The institution’s mission is clear and articulated publicly; it guides the institution’s operations.

Core Component 1.A. The institution’s mission is broadly understood within the institution and guides its operations.

Sub-component 2. The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.

HLC Criterion Three. Teaching and Learning: Quality, Resources, and Support The institution provides high quality education, wherever and however its offerings are delivered.

Core Component 3.E.The institution fulfills the claims it makes for an enriched educational environment.

Sub-component 1. Co-curricular programs are suited to the institution’s mission and contribute to the educational experience of its students.

YSU’s 2020 Strategic PlanYSU’s 2020 Strategic Plan

Office of Assessment (OOA)Office of Assessment (OOA)

The new University-wideStrategic Plan is organizedaround four cornerstones Student success The urban research

university mission/transition

Regional impact Institutional accountability

and sustainability

Two goals of the OOA (to increase campus engagement in assessment and to successfully demonstrate progress in the HLC Academy for the Assessment of Student Learning) are directly tied to 2 cornerstones of the 2020 Plan

Accountability and Sustainability which emphasizes shared responsibility.

Student Success since it documents student achievement in curricular and co-curricular areas.

The new Engagement in Assessment Portfolio will provide documentation.

Example: Strategic Plan Link12

YSU’s 2020 Strategic PlanYSU’s 2020 Strategic Plan

Office of Community InvolvementOffice of Community Involvement

Theme 2.2. Satisfaction with non-

academic experiences and campus life-including the importance of participation in service learning and community activities

The main goal of this unit is to increase participation in community activities as well as meaningful reflection on the impact of those activities on students’ development

Example: Strategic Plan Link13


2. Describe the goals for your unit from the previous academic/fiscal year

Example (from Office of Campus Planning and Community Partnerships):

to annually update the Campus Master Plan and undertake such studies as may be required to implement specific recommendations in accordance with the University’s annual capital improvement program;

to complete the Campus Roadway Study initiated by the City of Youngstown and focused upon the major arterial roadways serving the campus;

to implement improved bus service to the campus in collaboration with the WRTA and the University’s Alternative Transportation Community; and

to collaborate with Student Government to develop a matrix of student volunteer requirements and opportunities.

14


3. How do your goals support the learning environment at YSU?

Think big picture-not just the students, but also faculty, staff and the university as a system

Every unit contributes to the learning environment!

15

HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.

Example

Support Learning Environment Example (from the Reading and Study Skills Center):

The Reading and Study Skills Center primarily serves first-year students….this Center is the “first stop” in building a foundation for these students’ success and eventual retention for YSU. By building said skills, instructors and faculty should be experiencing students in their classrooms who are capable, responsible students, who read and think more critically than students who may not have been exposed to the concepts and skills taught in one of the RSS courses.

16


Describe your current assessment activities. 4. Provide a brief summary of unit and/or

program improvement data collected that document your unit’s assessment activities for the 2011-12 academic/fiscal year

Explain how you measured unit and/or program activities for continuous improvement

Does not need to be related to learning outcome, only focused on continuous reflection and improvement

Not data, but methods of collection (e.g., survey, performance evaluation)

17


Data collection summary:

18


Core Component 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.

Sub-component 2. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals. The institution assesses achievement of the learning.

Example

Current Assessment Activities Example (excerpt from Institutional Research and Policy Analysis):

Collected participation rate data on the CLA, NSSE, and the Graduate Plans Survey

Archived data requests from department and unit faculty and staff

19


5. How was that data analyzed and used by your unit? Does not have to be a formalized approach Ex: a discussion of survey data during a staff meeting

20



Sub-component 3. The institution uses the information gained from assessment to improve student learning.

Example

Data Analysis and Use Example (from Academic Advisors):

Survey responses were analyzed and reviewed. The data led to: Majors Exploration Fair was held to address concerns raised in

the survey indicating students were uncertain about their major.

A subcommittee formed to address needs of undetermined students

An advising syllabus was developed to address the students’ needs as a result of narrative comments from students on the survey

21

6. How did your unit utilize feedback from the OOA regarding last year’s assessment report?

o If this is your first time completing a plan or report, answer “first reporting year”

o This is an important part of completing a full assessment cycle/spiral/loop

Section 1: Current Unit Status22

Assessment process image credit: http://pdx.edu/studentaffairs/assessment-planning-and-practice.

Example

OOA Feedback Example (from Academic Advisors):

Based on feedback from OOA to develop a LO that could lend itself to direct method of measure, added a LO that included: Students will be advised in a timely manner Student will correctly identify their advisor (is being

considered for the future)

23

Section 2: Assessment Plan Development

7. State at least one learning outcome that is relevant

for your unit Keep to 1 (maybe 2 LO’s) Keep to 1 focused project (not a laundry list) What are you interested in?

24


4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.

1. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals.


Sources for LO: A particular project that your unit will be

focusing on in 2011-12 ORLO that staff from your unit develop ORYSU’s General Education learning outcomes ORThe Council for the Advancement of Standards

in Higher Education (CAS) General Standards OR

Learning Reconsidered

25


Developing a learning outcome:Framing Question: What do you want

students/staff/faculty to know and be able to do as a result of interacting with your office or program you offer?

Should be SMART: Specific, Measurable, Attainable, Realistic, Timely

Two common methods to writing a learning outcome: ABCD Model SVP Model

26

Learning outcome material excerpted from GLACUHO Webinar on Writing Learning Outcomes by Jeff Long, May 2010


ABCD Model of Learning OutcomesAudience-target of the outcomeBehavior-verb in future tenseCondition-activity causing learningDegree of Achievement (optional)

________ ________ ________ ________ Condition Audience Behavior

Degree

27


ABCD Model Examples________ ________ ________ ________

Condition Audience Behavior (Degree-optional)

As a result of student staff training, student staff will be able to list (4 of the 5) department’s values.

Through publicity and training, staff will correctly complete online timesheets (90% of the time).

As a result of this workshop, attendees will understand the components of a well-written assessment plan.

28


SVO Model of Learning OutcomesSubject = learnerVerb = cognitive processObject = content/knowledgeNote: removes condition (e.g. specific intervention);

somewhat more like academic learning outcomes.

Examples:Students will identify academic resources on

campus.Faculty will use DARS system in advising.

29


Using Verbs in Learning OutcomesAvoid “know,” “understand,” “think,” etc. –

how will you measure?Consider the level of learning you expect

from your students/staff/faculty—do you want them to comprehend? Apply knowledge?

Verb list that considers the level of learning (aka Bloom’s Taxonomy)--very helpful!: http://www.clemson.edu/assessment/assessmentpractices/refe

rencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf

30

Example

Academic Advisors Learning Outcomes Example:

1.Students can independently seek out resources to aid in evaluating their progress toward degree completion

2.Students will be knowledgeable of the curriculum requirements for their chosen major

31


8. How does this learning outcome fit your unit? Why did you pick this LO?

Example (for fictional Office of Community Involvement):

This is directly related to our unit’s purpose to

promote the benefit of community involvement

to our students

32


9. How does this learning outcome fit in with the YSU

2020 Strategic Plan? Consider how the learning relates to the larger goals of the

university

33

HLC Criterion One. Mission The institution’s mission is clear and articulated publicly; it guides the institution’s operations.

Core Component1.A. The institution’s mission is broadly understood within the institution and guides its operations.

Sub-component 2. The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.

HLC Criterion Three. Teaching and Learning: Quality, Resources, and Support The institution provides high quality education, wherever and however its offerings are delivered.

Core Component 3.E.The institution fulfills the claims it makes for an enriched educational environment.

Sub-component 1. Co-curricular programs are suited to the institution’s mission and contribute to the educational experience of its students.


Learning Outcome Relation to Unit Relation to YSU’s Strategic Plan

As a result of a unit publicity campaign, students will be more aware of the importance of community involvement

This is directly related to our unit’s purpose, to promote the benefit of community involvement to our students

Promoting the importance of community involvement helps to encourage participation, and the Regional Engagement Cornerstone outlines the importance of student involvement in the community to support quality learning experiences and positive community change (in particular Theme 1, Priority 1).

Staff demonstrate awareness of professional responsibilities of students serving in community organizations

By ensuring our staff are responsive to the needs of the community organization with which we partner…we are well positioned to expand our community service opportunities

By maintaining positive and responsive relationships with our community partners, we enable students to interact positively with community partners, and foster the collaborative relationships with community organizations outlined in the Regional Engagement Cornerstone (in particular Theme 1, Priority 1).

Example, Questions 7-9—how do they align?

34


Measuring your learning outcome10. What is the way (or ways) you will measure

the status and/or progress in this learning outcome over the 2012-13 academic year?

Note: consider tools that would incorporate both direct and indirect measures

A minimum of one tool but we recommend no more than two measures

An “exemplary” answer would include a direct measure

35

Direct MeasuresDirect Measures Indirect MeasuresIndirect Measures

Direct evidence of learning outcome performance:

Factual knowledge questions embedded in a survey

Presentations Performances Simulations Internship supervisors’

evaluations

Self-reported achievement or proxy evidence:

Surveys, group discussions, focus groups, exit

Interviews Employment

placement rates

Examples: Direct vs. Indirect Measures

36

Methods: Both Direct and Indirect

Method Indirect Direct

Minute paper after a workshop on diversity

Perceptions of moral dilemmas regarding diversity

Factual question on the definition of diversity

Telephone calls to a Unit

Caller’s satisfaction with technology

A qualitative analyses of callers knowledge of technology based on their questions

Survey after a training Teacher and learner satisfaction

Factual question on knowledge of workshop content

37

Tools may be direct or indirect, depending upon how they are used and their purposes:


11. What do you anticipate the outcome of your measures to be? Or, what do you hope to

find out?

This can be a check on whether you have chosen the right LO

Results do not need to be “positive” to be valuable

38

Example

Outcome Example (From Alumni and Events Management):

With our event changes we do receive positive and constructive feedback. That is what we are always looking to get from student and the public. No event is perfect. We are constantly trying to grow and develop our ceremonies and that is not possible without constructive feedback.

39


Close the Feedback Loop12. What are the major constituency groups

for your unit? Note: think not just within your unit or your

supervisory structure (internal stakeholders), but about students/faculty/staff, the larger university, the surrounding community, and beyond (external stakeholders).

40



Sub-component 4. The institution’s processes and methodologies to assess student learning reflect good practice, including the sub stantial participation of faculty and other instructional staff members.

Example

Constituency Groups Example : Internally: we share and seek

recommendations from all our staff as well as the Division VP

Externally: we share and seek recommendations students, campus units who do work on community involvement

University Honors and Scholars YSU’s Task Force on Civic Engagement WYSU Alumni Services

41


13. How and when do you plan to involve your major constituency groups in the assessment process

e.g. data sharing and recommendation gathering)?

Mention both the venue and frequency of meeting share with groups outside your unit use active ways rather than passive ways to share findings seek recommendations from groups inside and outside the

unit

42



Sub-component 3. The institution uses the information gained from assessment to improve student learning.

Example

Feedback Loop Example (from fictional Office of Community Engagement):

We will compile the results from our survey and share the information with the groups listed above through: monthly meetings with internal staff to update, present data, and solicit recommendations; soliciting a story about the results in the Jambar for students (spring 2013); and inviting other campus units to come in person for a presentation and discussion of the data (April 2013).

43

Section 3: Relation to HLC Criteria

14. (OPTIONAL FOR 2012) Please provide an example of evidence from your unit for the new HLC Criterion Five (Resources, Planning, and Institutional Effectiveness), Core Component 5A (The institution engages in systematic and integrated planning.), Sub-component 2 (The institution links its processes for assessment of student learning, evaluation of operations, planning, and budgeting.): Can help to begin familiarizing yourself with new HLC Criteria How will assessment activities you are planning impact your

planning and budget decisions? How does your work help contribute to YSU’s body of

evidence for this criterion?

44

Example

Aligning Assessment with Budget Example (from fictional Office of Community Engagement):

“While we didn’t think of it this way before reading this question, the 2011 survey results made it clear that more staff and financial resources needed to be devoted to a campus campaign. Through this assessment plan, we are devoting both fiscal and staff resources in order to carry out plans. Results from this year’s assessment will inform future planning and budgets.”

45

Section 4: Conclusion

Executive Summary15. Summarize the key ideas from Section Two

only Assessment Plan Development

one paragraph is fine a maximum length of one half page allows us to compile and develop themes from 40+

units

46

Section 4: Conclusion

16. (Optional) Is there anything else you would like to share regarding your assessment plan and/or is there any particular area on which you would like assistance or feedback?

This is a chance to share something that didn’t “fit” with our questions but is important to your assessment “story”

Or, it is something about which you could use our help or advice

47

Examples

We need help in having a forum to share our data from our prior year’s report. Can you help us set up a workshop or forum?

We are not certain how to incorporate a direct measure with this learning outcome-do you have any ideas

48

Example

We would love some examples of rubrics that have already been developed to measure community engagement. Do you have some instruments to share with us

We would like to use the institutional account for survey monkey. How do we go about learning more about using this survey?

49

New Online Reporting Format

Online Assessment Plan Submission Form: www.jotform.us/ysuassessment/2012naplan

50

Thank you for your participation!

51

Documents

OFFICE OF ASSESSMENT JULY 11, 2012 Non-Academic Assessment: Developing an Assessment Plan