Amber MalinovskyAssistant Director

Institutional AssessmentTexas A&M University

Texas A&M University, College Station, Texas 48,000 students 2,700 faculty 5,500 staff 5,200 acres

WEAVE users and programs 470+ programs in entity tree 700 WEAVE users

Office of Institutional Assessment Four full-time employees, two graduate assistants Variety of duties that support assessment Liaison structure• Ten colleges• Two branch campuses (Galveston and Qatar)• Multiple support units (student affairs, undergraduate

programs, etc.)

Choose a topic Often suggested by users/Assessment Liaisons Sometimes specific to audience (academic vs. support, by

discipline, etc.)

Conduct research and collect examples

Draft and revise slides and materials Our “simple” formula: definition + tips + examples + cautions

+ putting it all together = meaningful, measurable, manageable

Choose dates and locations

Advertise and track registrants Announcement sent on email lists and posted on website Online form for registration

Evaluation and follow-up

Past workshops: http://assessment.tamu.edu/wkshp_pres/index.htm

Dr. Loraine PhillipsAmber Malinovsky

Part One Introduction to Assessment Components of an Assessment Plan

Mission Outcomes Measures Achievement Targets

Components of an Assessment Report Findings Analysis of Findings Action Plans Annual Reporting

May 31, 2009 Data collected August 1, 2009 Findings entered September 1, 2009 Analysis Questions answered,

along with any appropriate action plans

September 15, 2009 Annual Report question answered

October 1, 2009 2008-2009 Cycle Year CLOSES no more editing

SACS Comprehensive Standard 3.3.1

3.3 Institutional Effectiveness3.3.1 The institution identifies expected outcomes, assesses the

extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)

3.3.1.1 educational programs, to include student learning outcomes

3.3.1.2 administrative support services

3.3.1.3 educational support services

3.3.1.4 research within its educational mission, if appropriate

3.3.1.5 community/public service within its educational mission, if appropriate

Adapted from: Trudy Banta, IUPUI

The mission statement links the functions of your unit to the overall mission of TAMU.

A few questions to consider in formulating the mission of your unit:

What is the primary function of your unit? What core activities are involved? What should those you serve experience after

interacting with your unit?

Brief, concise, distinctive Clearly identifies the program’s purpose Clearly aligns with the mission of the division and the

University Explicitly articulates the essential functions/activities of

the program Clearly identifies the primary stakeholders of the

program: i.e., students, faculty, parents, etc.

“University Career Services, an integral part of the educational

process, assists students and alumni in assessing their career

possibilities, setting their personal goals and achieving their

objectives toward becoming productive citizens in the global

community. While assisting its clients in identifying professional

employment opportunities, University Career Services also

provides the university community with insights into the ever-

changing world of work to help develop realistic ways to better

educate tomorrow’s leaders.” (Texas Christian University)

When writing Learning Outcomes, the focus must be on the stakeholders (e.g., students, faculty, staff, and others) and what they will think, know, do, or value following the provision of the service.

Process statements Relate to what the unit intends to accomplish• Level or volume of activity• Efficiency with which you conduct the processes• Compliance with external standards of “good practice in the field” or

regulations

Satisfaction statements Describe how those you serve rate their satisfaction with your unit’s

processes or services

Access Accountability Affordability Climate Educational

Effectiveness/Quality

Facilities & Equipment Faculty Financial Support Student Preparation Productivity Technology

Consider such questions as: What are the most important results or impacts that

should occur as a result of your unit’s activities?

What are your critical work processes and how should they function?

What does the end user experience through interaction with your unit?

Learning Outcomes Students receiving advising services will be able to identify and utilize

campus resources. Staff and faculty will be able to use the Oracle system for purchasing.

Process statements The number of faculty training workshops and participation rates for

WebCT will relate to the needs. The travel office will promptly process travel requests.

Satisfaction statements Faculty will report satisfaction in the training received for WebCT. Faculty and staff will report satisfaction with travel processing under the

Oracle system.

Outcomes should be: linked to the unit’s mission realistic and attainable limited in number (manageable) something that is under the control of the unit measurable and/or observable meaningful target key services or change points use action verbs

After establishing your outcomes…

Define and identify the sources of evidence you will use to determine whether you are achieving your outcomes.

Detail what will be measured and how Identify or create, if necessary, measures that help your

unit in making critical decisions about its processes and services.

Some things to think about:

How would you describe the end result of the outcome?

How will you know if this outcome is being accomplished?

What will provide you with this information?

Where are you currently delivering the outcome?

Are there any naturally occurring assessment opportunities?

What measures are currently available?

  Learning Outcome Method of Delivery How, When, and Where Information will be

Gathered

Outcome 1        

Outcome 2        

Outcome 3        

Etc.        

Measurable and/or observable

You can either observe it, count it, quantify, etc.

Meaningful

If captures enough of the essential components of the objective to represent it adequately

Manageable

It can be measured without excessive cost or effort

Direct measures are those designed to directly measure:

what a stakeholder knows or is able to do (i.e., requires a stakeholder to actually demonstrate the skill or knowledge)The benefit of programming or intervention

Participation data

Observation of behavior

Culminating experiences (e.g., presentation, project, internships, etc.)

Collection of work samples (portfolios)

Pre- and post-measures

Volume of activity

Level of efficiency (average response time)

Measure of quality (average errors)

Indirect measures focus on:

stakeholders’ perception of their level of learningstakeholders’ perception of the benefit of programming or interventionstakeholders’ satisfaction with some aspect of the program or service

Surveys

Exit interviews

Retention/graduation data

Demographics

Focus groups

An achievement target is the result, target, benchmark, or value that will represent success at achieving a given outcome.

Achievement targets can be specific numbers or trends.

Students will achieve a score of 3 or 4 on the rubric.

Average score on rubric is a 3 or better (on a scale of 0 to 4).

90% of the transcripts will be sent within three days.

Each employee will participate in a minimum of two training/development programs per year.

Acquisition statistics will indicate growth in the overall collections that support academic programs.

Outcome: Improve and strengthen media relations and publications

Measure 1 (Direct): Track media pitches/releases.

Achievement Target: Average of 250 media pitches/releases for the year.

Measure 2 (Indirect): Conduct annual readership survey of university magazine.

Achievement Target: Feedback via readership survey will indicate average ratings

of 7 or better on a 1-10 scale.

Outcome: To provide a library web site that enables users to locate and use information on their own

Measure 1 (Direct): Usability testing with small groups

Achievement Target: Participants in usability testing demonstrate web site’s ease of use

Measure 2 (Indirect): LibQUAL+ assessment program

Achievement Target: LibQUAL+ results indicate faculty and student satisfaction

Writing your Achievement Target as part of your Outcome

Example:

• Outcome: 80% satisfaction rate from instructors

• Measure: Biannual instructor satisfaction survey

• Achievement Target: 80% satisfaction rate

Possible Steps to Revision:

• Satisfied with what?

• New Outcome: Instructors will report satisfaction with the resources available for course development.

Having the same Measure and Achievement Target Example 1:

• Measure: Annual review of forms

• Achievement Target: Annual review of forms

Possible Steps to Revision:

• Go back and look at outcome. Outcome states that Forms and paperwork will be clear, concise & accurate.

• If the Annual review shows that they are not clear, concise & accurate, what will need to happen?

• New Achievement Target: Decrease in number of revisions needed to make forms more understandable and accurate.

Having the same Measure and Achievement Target

Example 2:

• Measure: 50 slots for scholarship recipients secured

• Achievement Target: 50 slots secured for scholarship recipients

Possible Steps to Revision:

• Remember: Measures are more general. Achievement Targets are specific.

• Identify what it is that you’re doing to determine whether or not you’ve met the achievement target. THIS is your measure.

• New Measure: Track number of scholarship recipients.

Writing your Achievement Target as part of your Measure

Example:

• Measure: Score a 3 or 4 on an identified technology assignment

Possible Steps to Revision:

• Remember: Measures are more general. Achievement Targets are specific.

• Outcome states: Demonstrate competency in technology

• Revise Measure to read: Identified technology assignment.

• Add Achievement Target: Students will achieve a 3 or 4 on the rubric.

Assessment Plan Mission/Purpose Outcomes, learning & program Assessment Methods with

achievement targets

Assessment Report Mission/Purpose Outcomes, learning & program Assessment Methods with

achievement targets Findings Action Plans Analysis Questions Annual Report Section

SACS Comprehensive Standard 3.3.1

3.3 Institutional Effectiveness3.3.1 The institution identifies expected outcomes, assesses the

extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)

3.3.1.1 educational programs, to include student learning outcomes

3.3.1.2 administrative support services

3.3.1.3 educational support services

3.3.1.4 research within its educational mission, if appropriate

3.3.1.5 community/public service within its educational mission, if appropriate

and provides evidence of improvement based on analysis of the results…

Findings = Assessment data

On an Assessment Report, “findings” refers to a concise summary of the results you gathered from a given assessment measure.

The language of this statement should parallel the corresponding achievement target

Describe your results in enough detail to prove whether you have met, partially met, or not met your achievement target.

It is not necessary to provide any interpretation of your data in your findings summary.

(Optional) Attach documents to support your data. These can include survey instruments or results, reports, committee members and minutes from meetings, etc.

Example 1:

Achievement Target: Overall mean score of students from program meets or exceeds state average score.

Findings: The overall mean score of students from the Teaching, Learning, and Culture program exceeded that of the state average score of the state certification exam. Results: Program overall mean scaled score—91.50, State overall mean scaled score—79.13.

Example:

Achievement Target: Decrease by 40% the number of undeclared students from the entering major (Fall 2008) to current major (Fall 2009)

Findings: 34% of undeclared students who had registered for Fall 2009 classes by the beginning of June had declared a major.

Example:

Achievement Target: 10 campus-wide workshops conducted in November and December of 2008.

Findings: 8 campus-wide workshops were conducted in November and December of 2008.

Reflect on what has been learned during an assessment cycle

Identify areas that need to be monitored, remediated, or enhanced

Three key questions are at the heart of your analysis:

What did you find and learn?

So What does that mean for your academic program or support unit?

Now What will you do as a result of the first two answers?

You will want to reflect on the following areas: Student Learning Outcomes, if applicable

Program Outcomes, if applicable

The Assessment Process

After reflecting on the findings, you and your colleagues should determine appropriate action to improve the program. This will lead to at least one action plan.

Actions outlined in the action plan should be specific and relate directly to the outcome and the results of assessment.

We do not recommend having an assessment report without any action plans.

SACS 3.3.1

We do recommend that you keep the number of action plans manageable.

The final step in your assessment report is to consider how your program contributed to one or more of the following:

Departmental goals or imperatives

College goals or imperatives

University mission, goals or imperatives

Assessment Plans Mission/Purpose Statements Outcomes Measures and Achievement Targets

Findings/Report Achievement Targets Findings Analysis Action Plans

You do not have to assess everything every year Modify something already being done that is

meaningful to the program Summarize your Findings Limit your Action Plans and keep them manageable Consider how your program fits within the university’s

mission Be flexible—this is an iterative process

The Principles of Accreditation: Foundations for Quality Enhancement. SACS COC. 2008 Edition.

Banta, Trudy W., & Palomba, C. (1999). Assessment Essentials. San Francisco: Jossey-Bass.

Banta, Trudy W. (2004). Hallmarks of Effective Outcomes Assessment. San Francisco: John Wiley and Sons.

Walvoord, Barbara E. (2004). Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. San Francisco: Jossey-Bass.

Assessment manuals from Western Carolina University and Texas Christian University were very helpful in developing this presentation.

Putting It All Together examples adapted from Wright State University Libraries Assessment Plan, UHCL President’s Office Assessment Plan

North Carolina State University http://www2.acs.ncsu.edu/UPA/assmt/admin_examples_others.htm

University of Central Florida—Handbook for Admin Units http://oeas.ucf.edu/doc/adm_assess_handbook.pdf

Texas Christian University http://www.assessment.tcu.edu/

Each Other OIA consulting

Amber Malinovsky, Assistant DirectorOffice of Institutional Assessment

Questions?

What was the most valuable thing you learned?

What is one question that you still have?

What do you think is the next step that your program needs to take in order to implement systematic program assessment?

Our Staff Dr. Loraine Phillips, lhphillips@tamu.edu

Amber Malinovsky, amalinovsky@tamu.edu

Our Website http://assessment.tamu.edu

TAMU Assessment Conference Website http://assessment.tamu.edu/conference/index.html

February 21-23, 2010College Station, Texas

http://assessment.tamu.edu/conference

Registration now open!

Plenary Speakers:Dr. Belle S. WheelanDr. Clifford Adelman