Assessment of research: Where do we go from here? Linda Rueckert, Northeastern Illinois University (lruecker/assess.htm) Kathleen

Assessment of research: Where do we go from

here?

Linda Rueckert, Northeastern Illinois University(http://www.neiu.edu/~lruecker/assess.htm)

Kathleen Morgan, Wheaton College

Who are the stakeholders in undergraduate research assessment?

*Institutions/agencies

*Departments and Programs

*Faculty

*Students

Stakeholders in Undergraduate Research: why assess?

Institutions

*retention

*grants-seeking

*grant stewardship/accountability

How you ask will depend greatly on what you want to know, and why…


Departments/Programs

*program improvement



Faculty

*training improvement

*are they learning what you think you’re teaching?

*(n=61 CUR attendants)

Top 3:

1. critical thinking2. communication skills3. applied knowledge/disciplinary literacy



Students

*improve grad school prospects

*improve employability

*fun

*develop closer relationships with faculty

*we’d like to find out more! (see survey on yellow handout)—and you can help!


Stakeholders in Undergraduate Research: how assess?

Institutions

*Oak Ridge Associated Universities: custom-made survey instruments

*University of Michigan—structured interviews(http:///www.umich.edu/~urop/facres/evalurop.html)

*University of Delaware Research-based Education: (http://www.udel.edu/RAIRE)

A few examples… ..can you provide others?


Departments/Programs

*Northern Illinois University Psychology Dept.


NEIU capstone analyses

• Students do independent research, field experience, or enhanced class as capstone project.

• All are required to write a final APA-style paper.

Capstone scoring rubric

• Content– Theoretical basis– Breadth and depth– Creativity– Design– Statistics– Conclusions– Relate to intro.– Broader

implications

• Style– Flow– Spelling & grammar– Clarity– APA format

Capstone scores for content and style (old data)

22.22.42.62.8

33.23.43.63.8

4

Score

Content Style

ClassIndependent

Capstone scores on content subscales (old data)

2

2.5

3

3.5

4

Score

Theor.basis Breadth Creativity Concl. Relates Implications

Capstone scores for content and style (new data)

2

2.2

2.4

2.6

2.8

3

3.2

3.4

Score

Content Style

Class Exp.Independent

Capstone scores on content subscales (new data)

2

2.5

3

3.5

4

Score

Class Exp.Indep.


Faculty

Critical Thinking Assessment Tools:

*Watson Glaser Critical Thinking Appraisal RANRA

*California Critical Thinking Skills Test

*Tim van Gelder’s Critical Thinking Assessment Site (lots of links to various tools here): http://www.austhink.org/critical/pages/assessing.html


Communication Skills Assessment:

*Liverpool Communication Skills Assessment Scale (LCSAS)

*Biography of Assessment Alternatives: Oral Communication. Innovative Assessment. ERIC document ED 422595

*Assessment of Basic Oral Communication Skills: A Selected, Annotated Bibliography. ERIC document ED 319086


Students

*Kardash (2000). Self-rating instrument

--used a 1-5 scale, asked research interns to rate themselves on items like

“To what extent do you feel you can……

…understand contemporary concepts in your field? …make use of the primary scientific literature in your field? …formulate a research hypothesis based on a specific question?”, etc.

*reflective judgment essays

*goal setting and ranking

*our student survey


LEARNING FROM YOUR RESEARCH EXPERIENCE*

Before you begin your research work, take a few minutes and complete the statements below. This will help you to begin thinking about your learning goals for participating in research. Although at this point, you can only speculate about the research experience, the more you think about it now, the more you will learn from it.

A. I am engaging in research because:

B. My goals for participating in research include:

C. I think I will be performing the following duties/tasks:

D. I hope to develop the following skills:

E. In addition to testing an hypothesis, I think I might learn about:

F. What skills or perspectives do you think you already have to contribute to your research experience?

Turn your completed handout in to your supervising faculty member.

Sample goal setting document:

Principle Investigator (PI) Self-EvaluationStudent Name:__________________________________ Semester_______ Yr_________Title of research project you worked on:____________________________________________________________________________________________________________________________Estimated # of hours spent on this project during the entire semester____________________

DIRECTIONS: The academic credit that you will receive upon satisfactory completion of your role as a PI reflects the college’s understanding that learning here also takes place outside the conventional classroom. To that end, it is useful to think about just what it is that you have learned. Using the guidelines below, write a 1-3 page reflection essay on your experience. You may find it useful to refer to your research journal as you write. Attach your essay to this cover sheet and return it to Dr. Morgan.

I. Briefly describe the research project you developed or are developing and your primary responsibilities in that development. What did you do (or what do you plan to do)? What have you found out?

II. Reflect (write about what you have learned):

*Some of the skills that you have developed and how you learned them*What you have learned from your supervising professor, your research assistants (if any), or others who might have been involved in the project’s development*What you expected to do and what you actually did*The relationship, if any, between this out-of-the-classroom learning experience and your academic work

III. Have you changed as a result of this experience? If so, in what ways?

IV. What grade do you think you should receive for your efforts? Why?

Sample guided reflective essay:

A Few Last Words on Assessing Undergraduate Research Experiences

If you are just beginning an assessment program—whether of your classroom or your research program—here are a few suggestions to keep in mind (courtesy of Tom Angelo again):

1. Know what it is you want to know, BEFORE you begin planning your assessment. Do you want to know if student researchers go on to graduate schools more often than students who do not participate in research? Do you want to know if your student researchers are sufficiently trained in execution of a protocol to be able to manage it on their own? Do you want to know if student researchers' self-confidence increases after their experiences? Knowing what outcome you want to assess is essential to designing and implementing the right assessment.

2. Keep it simple and short. Don't make assessment into a chore or burden for either you or your students. Collect only as much data are you can reasonably use and respond to.

A Few Last Words on Assessing Undergraduate Research Experiences

3. Keep it focused. (This harkens back to knowing what you want to know!) While there are many interesting questions you might ask, focus on those most likely to improve the achievement of your own teaching, research, or program goals. Make sure that your students understand the purpose and value of your assessment. This too is an important component of assessment that we often forget.

4. Don't ask if you don't want to know, or won't be able to respond.

5. Make sure your aim is true. ALWAYS try out a new assessment technique on a colleague before trying it out on your students.

6. Close the feedback loop. Make sure to let students or other stakeholders know the outcomes of the assessment and what changes (if any) those outcomes suggest for you and them.

7. Let us know about your assessment program!

Send your assessment ideas to:

[email protected]

Or

[email protected]

Thanks!

Documents

Assessment of research: Where do we go from here? Linda Rueckert, Northeastern Illinois University (lruecker/assess.htm) Kathleen