UNDERGRADUATE RESEARCH AT THE UNIVERSITY OF VIRGINIA · 2017-02-17 · Given this understanding, the assessment of undergraduate research competency would have to provide for differences

UNDERGRADUATE RESEARCH

AT THE UNIVERSITY OF VIRGINIA A Core Competency Assessment Report

Submitted by the Undergraduate Research Assessment Committee

Coordinated by the Office of Institutional Assessment and Studies Report prepared by IAS staff (Lois Myers and Sarah Schultz Robinson)

December 2013

Undergraduate Assessment Committee:

Handler, Richard (CLAS, Anthropology) – CO-CHAIR

Wolcott, David (Provost) – CO-CHAIR

Berne, Rosalyn (SEAS, STS)

Brickhouse, Anna (CLAS, English)

Culaty, Brian (Provost)

Freedman, Paul (CLAS, Politics)

Friberg, Elizabeth (NURS)

Kelly, Luke (SED, Kinesiology)

Marshall, Paxton (SEAS, ECE)

Maxham, Trey (Assoc. Dean, SCC)

Mills, Aaron (CLAS, Environmental Science)

Palmer, Michael (TRC; CLAS, Chemistry)

Plasket, Donna (SCPS, BIS)

Pocanic, Dinko (CLAS, Physics)

Rasbury, Michael (CLAS, Drama)

Roettger, Betsy (SARC, Architecture)

Russell, Lucy (Provost)

Soule, Kathy (Library)

Institutional Assessment and Studies Core Competency Research Assessment 2012

2

TABLE OF CONTENTS

Executive Summary 1 I. Assessment Process and Methodology 2 II. Summary of Results 7 III. Findings 14 IV. Recommendations 15 Appendices 16

A- SERU Survey Results- Participation 17 B- Rubric for Academic Research Project 19 C- Rubric for Applied Research Project 21 D- Modified Research Project Rubrics 24 E- Results by Discipline Area 30 F- SERU 2012 Survey Results-Engagement, Learning 65

1

EXECUTIVE SUMMARY

In 2008-2009, the University’s Office of Institutional Assessment and Studies initiated planning for the first institution-wide assessment of undergraduate research competency. The Undergraduate Research Competency Assessment Committee defined research as “the practice of carefully formulating or addressing a question, problem or objective, analyzing it within a disciplinary or interdisciplinary framework, producing findings, conclusions, designs, or creative works, and clearly communicating and defending such to a critical audience.” Eight learning outcomes closely reflected the definition: Students engaging in undergraduate research will:

1. Approach the study from a particular disciplinary or interdisciplinary point of view 2. Form a thoughtful research question or identify a problem to be solved 3. Use a variety of sources 4. Collect pertinent information 5. Analyze data/evidence collected 6. Draw logical and defensible conclusions 7. Communicate clearly and effectively their findings and conclusion 8. Defend their research to a critical audience.

Acknowledging the variability in disciplinary or interdisciplinary frameworks for conducting research, two scoring rubrics were made available for assessment of student research papers or projects with permission to modify the rubrics to meet specific disciplinary priorities. Throughout 2010-12, 332 research papers and projects, representing nine major disciplines, were assessed. Overall, research papers and projects submitted by students met the standards for competence in research with half assessed as highly competent. Students demonstrated the most competence for three outcomes: approaching the study from a particular disciplinary or interdisciplinary point of view, collecting pertinent information, and communicating clearly and effectively. They demonstrated the least competence for three outcomes: forming a thoughtful research question or identifying a problem, analyzing their data or evidence, and drawing logical and defensible conclusions. The learning outcome, Question (Form a thoughtful research question or identify a problem), was most likely to distinguish highly competent from less competent research, suggesting an area for additional focus in guiding student research. This initial assessment of undergraduate research posed multiple and significant challenges: the multi-disciplinary environment in which research practices differ widely by discipline; the fact that research is a “mega-competency” requiring students to apply two to five multiple core competencies (critical thinking, writing, quantitative reasoning, scientific reasoning, and/or oral communication); the special challenge of assessing design and creative research assignments; and the considerable task of assessing each student paper according to seven or eight outcomes, many of which are complex enough that they could support their own separate rubrics.

As this was the first University-wide assessment of a complex competency, numerous challenges to process and reliability of results arose. Recommendations focus primarily on means to address those challenges for the next assessment of this competency.


2

I. ASSESSMENT PROCESS AND METHODOLOGY In 2008-9, the University initiated the first institution-wide assessment of undergraduate research competency. The State Council of Higher Education for Virginia (SCHEV) requires that all institutions of higher education assess six undergraduate competencies, generally one each year. SCHEV permits institutions to replace one of the competencies. The University did so, selecting undergraduate research to replace technical literacy. The assessment of undergraduate research was conducted for two purposes: to comply with SCHEV requirements to conduct assessments of undergraduate competencies and to provide evidence of student competency for the Quality Enhancement Plan. The goal of the QEP was to enhance student-faculty engagement by increasing undergraduate research and academic public service opportunities. The QEP assessment plan incorporated both direct assessment of competency as well as indirect evidence regarding student participation in and experience with research. The University does not mandate that all undergraduates conduct or participate in research, although many degree programs do. Nearly all fourth-year students (94%) in 2012 reported that they were currently participating in or had done “a research project, creative activity, or research paper as part of coursework.” Half of fourth-years reported having completed a “significant research project”1 during their tenure at U.Va. A quarter had assisted faculty in research for credit, fewer for pay or as volunteers. Fourth-years reported that the opportunity to learn research methods was important to them. (Appendix A) Two faculty committees, operating sequentially, guided the assessment planning and interpretation of results. The first committee (2008-9) laid the groundwork for the assessment, agreeing on the overall learning goal, definition of research, and the learning outcomes. The learning goal was to:

Develop in students the capacity to conduct research, and the habits of mind of learners who frame and contextualize knowledge within their own reasoned points of view, recognize that knowledge evolves, and actively rethink and question existing assumptions, arguments, methods of inquiry, and facts.

Within that goal, students are expected to learn to approach the study of a subject or problem from a particular disciplinary or interdisciplinary point of view and to apply the techniques or methods of that discipline in conducting research. The committee subsequently defined undergraduate research as:

the practice of carefully formulating or addressing a question, problem or objective, analyzing it within a disciplinary or interdisciplinary framework, producing findings, conclusions, designs, or creative works, and clearly communicating and defending such to a critical audience.

While the original assessment plan proposed to examine the overall research competency of undergraduates, the Office of Institutional Assessment and Studies, in concert with the committee, developed an assessment plan that would both assess overall competency and allow for differences in discipline-specific research approaches and expectations of students. A recent study of undergraduate

1 Defined as follows: “By significant research project, we mean one in which you: either individually or within a

group a) carefully formulated or addressed a question, problem or objective, b) analyzed it within a disciplinary or interdisciplinary framework, c) produced findings, conclusions, designs, or creative works, and d) clearly communicated and/or defended your work to a critical audience.”


3

research had highlighted the variability in disciplinary approaches to undergraduate research2. Focusing on the research that fourth-year students typically conduct, the researchers interviewed and surveyed undergraduates from 15 majors. They concluded that disciplinary differences in both purpose and method were distinctive, even down to the level of the major (e.g., academic programs within the social sciences differ in methods; science and math, often considered together, employ different approaches, etc.). Examining critical thinking, a core skill essential to conducting research, Nilson came to much the same conclusion: the methods of the disciplines differ remarkably: the physical and biological sciences, the social sciences, the history-based disciplines, and literary scholarship all use different approaches.3 Given this understanding, the assessment of undergraduate research competency would have to provide for differences in approach. Adding another layer of complexity, compared to the other five undergraduate core competencies4, undergraduate research is a “mega-competency.” To conduct research, students apply multiple core competencies. To complete a research project, a student will certainly draw on critical thinking skills and communication skills—writing, oral, or both. Within the sciences, social sciences, commerce, engineering, or nursing, however, students may also need to rely on their abilities in quantitative and/or scientific reasoning. As such, the assessment of research would entail assessment of two to five other competencies as well, depending on the discipline.

Within this context and recognizing that undergraduate research is defined by the disciplines, the committee identified four distinct types of research approaches—academic, applied, design, and creative—and proposed that rubrics be created to address each of the four types of research. These rubrics would be employed by programs in conducting assessments. While definitions for the four types of research projects were proposed, subcommittees were successful in developing two scoring rubrics to apply (Academic Research and Applied Research). (Appendices B and C) Eight learning outcomes reflected in the rubrics were stated as:

Students engaging in undergraduate research will: 1. Approach the study from a particular disciplinary or interdisciplinary point of view 2. Form a thoughtful research question or identify a problem to be solved 3. Use a variety of sources 4. Collect pertinent information 5. Analyze data/evidence collected 6. Draw logical and defensible conclusions 7. Communicate clearly and effectively their findings and conclusion 8. Defend their research to a critical audience.

2 Jon Peterson, Catharine Beyer, Sophia Chang, Alaina Giesbrecht. The UW Senior Research Study 2009. Office of Educational Assessment. Retrieved from http://www.washington.edu/oea/pdfs/reports/OEAReport1001.pdf . 3 Nilson, L. (2010). Teaching at its Best: A Research-Based Resource for College Instructors (3

rd edition). San

Francisco, CA: Jossey-Bass. 4 Critical thinking, quantitative reasoning, scientific reasoning, written communication, and oral communication

http://www.washington.edu/oea/pdfs/reports/OEAReport1001.pdf


4

As the first institution-wide assessment of research competency and without previous assessments to serve as precedent, the standards adopted in previous core competency assessments would apply:

25 percent of research projects would be highly competent (avg. total score=75% of possible points)

75 percent of research projects would be competent or better (50% of possible points)

90 percent of research projects would be minimally competent or better (33% of possible points)

Throughout 2010-11, the two scoring rubrics were recommended to academic programs to use in conducting assessments. Along with guidance regarding application of scoring rubrics, programs were advised that they could amend either rubric to meet specific program needs. Several programs in three schools (Commerce, CLAS, SON) assessed student research projects by having instructors apply a rubric; three of those programs modified the rubric by adding, subtracting, or re-stating outcomes (Appendix D). Not all major disciplines were represented in the assessments received, however. The second committee (2011) solicited student research papers and projects for assessment from disciplines that had not adopted or applied either of the rubrics. Program faculty were asked to identify and submit student research projects where students had been asked to complete the full process of research, from identification of a research question/problem through data collection and analysis and closing with conclusions and the write-up. In response, nine programs in four schools (SCPS/BIS, CLAS, SEAS, Education) submitted student papers or projects. IAS facilitated the assessment of these research papers. The effort was not successful, however, in soliciting and assessing student research from the fine arts or from Architecture. In these fields, the primary product is not likely to be a written document (although an explanatory written report may accompany the final product), but rather a performance, event, or presentation. That said, students in the fine arts or Architecture do undertake systematic research in developing their creative projects or designs. This research is incorporated into the design process or into the preparation for performance or gallery show, but it is not as readily accessible or assessable through rubrics. In all, 332 assessments of research papers or projects, representing nine major disciplines, were conducted (Figure 1). The assessments included:

BIS/SCPS- 20 capstone projects

Humanities/College- 35 papers from three programs

Interdisciplinary/College- 46 papers from one program

Science/College- 61 papers from two programs

Social Science/College- 56 papers from two programs

Commerce- 48 student team capstone projects

Education- 20 research proposals (students are not asked to conduct research, but are required to prepare research proposals to demonstrate their knowledge of the research process)

Engineering- 36 research projects (one component of thesis portfolio)

Nursing- 10 Distinguished Majors theses


5

Figure 1. Assessments by Discipline

This report describes the process, results, and findings from the assessment of the research. The report closes with a set of recommendations. In 2010-11, IAS invited faculty and selected graduate students to participate in scoring sessions to assess the solicited research papers and projects by applying the Academic Research Project Rubric. For papers from the College, scorers were asked to read and rate research papers both from their own disciplines and from other disciplines. Scoring sessions began with a norming exercise to assure that evaluators were in accord in the application of the rubric. Papers were stripped of student identification and randomly assigned to evaluators. Evaluators rated each paper according to the learning outcomes contained in the scoring rubric, giving each outcome a rating from 1 (not competent) to 5 (highly competent). Two evaluators scored each student paper, and the two evaluators’ scores for each outcome and for the overall rubric were averaged to create scores for each paper. Rubric results were analyzed across disciplines (n=332) and by discipline (Appendix E). As some programs had altered the rubric, applying more or fewer outcomes or by using a scale from 1 to 4, the scores were analyzed in terms of percent of possible points earned. Inter-rater reliability of ratings was low to moderate for the assessments. For the College assessments and for the SEAS assessment, reliability of ratings was low, ranging from 0.20 to 0.36 (Table 1). For the other schools, reliability of ratings was moderate (range .46-.83). (Reliability data were unavailable for the assessments in the sciences as one program used only one rater per project.)

3%

6%

6%

11%

11%

14% 14%

17%

18%

Nursing

BIS/SCPS

Education

Humanities/CLAS

DES/SEAS

Interdisc/CLAS

Commerce

Social Science/CLAS

Science/CLAS


6

Table 1. Reliability of Ratings by Discipline

α5 % differing by

>1 point Raters

BIS/SCPS 0.46 26% Program faculty

Humanities/College 0.36 29% Mixed group

Interdisciplinary/College 0.36 0% Program faculty

Social Science/College 0.17 25% Mixed group

Commerce 0.61-0.83 14% Program faculty

Education 0.53 19% Program faculty

Engineering 0.20 38% Mixed group

Nursing 0.64 18% Program faculty

N 271

The percent of reviewers’ ratings that differed by more than one point served as another measure of reliability. Given the complexity of this assessment task, we allowed at least ten percent of ratings to differ by more than one point. Only one of the assessments met this target. The percent of ratings that differed by more than one point generally ranged from 14 to 38 percent, suggesting considerable differences of opinion among reviewers. Reliability of ratings tended to be lower where the assessments employed reviewers from both within and outside the discipline (mixed group) and higher where the assessments were conducted only by faculty from the programs. Despite norming sessions designed to arrive at a shared set of expectations, it appears that reviewers were not able to fully overcome their own disciplinary expectations or understand the approaches of other disciplines. Indirect evidence was gathered through both surveys and structured interviews with students who had completed research projects. In 2012, all undergraduates were invited to take the SERU (Student Experience in the Research University) survey; of the 4710 who responded, about 1400 were fourth-years. In addition 17 fourth-year students who reported having completed a “significant research project” were interviewed in depth about their experience.

5 Cronbach’s alpha (α) is a measure of internal consistency; positive, higher values are interpreted to reflect

greater consistency.


7

II. SUMMARY OF RESULTS Study sample The assessment represented the range of disciplines, except for the important omission of architecture and the fine arts. Almost a third of the projects assessed were completed by teams of students. Other projects represented a wide range in assignments, from research proposals to seminar papers to thesis projects to Distinguished Major research projects (Table 2). In some cases, the assignments were relatively new to the program and still evolving while others were well-established requirements in the programs. In all cases, the research projects or papers were rooted in the practices of the individual disciplines.

Table 2. Number of Assessments by Discipline and by Type of Assignment

Seminar

paper Capstone/Thesis DMP Thesis Research Proposal

Laboratory Course

BIS/SCPS 20

Humanities 35

Interdisciplinary 46

Sciences 9 52 (team projects)

Social Sciences 56

Commerce 48 (team projects)

Education 20

Engineering 36

Nursing 10

Overall performance Papers and projects submitted by students met the standards for competence in research: 52 percent were highly competent, 94 percent were competent or above, and 99 percent were minimally competent or above (Table 3). Overall, the papers earned 76 percent of total points possible. A solid 40 percent of projects earned over 80 percent of possible points (Figure 2). One paper was judged not competent; it was from a program that had recently initiated the required assignment.

Table 3. Research Competence

Minimally Competent or above

Competent or above

Highly Competent

All Papers and Projects 99% 94% 52%

Standards for Competency 95% 50% 25%

N 331 312 173


8

Figure 2. Percent of Research Projects by Percent of Points Earned*

* One percent of projects were assessed as not competent. Performance by Discipline With one exception, each discipline met or exceeded the standards for performance (Table 4). The one discipline that did not meet the standard for percent highly competent missed the target slightly, which could be attributed to the use of scorers from outside the discipline (see discussion regarding reliability). In two disciplines—Commerce and the Sciences, four out of five projects were determined to reflect high competence (Table 4, Figure 3). Two other disciplines (Engineering and Humanities) showed the lowest proportions of highly competent papers; the lower performance can be attributed to the newness of the assignment in Engineering and the low reliability among the mixed group of raters for the Humanities papers.

0.6% 0.3%

20.8% 18.7% 19.6%

22.9%

17.2%

0%

10%

20%

30%

40%

31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100%

Pe

rce

nt

of

Re

sea

rch

Pro

ject

s

Percent of Total Points Earned

52% Highly Competent

42% Competent

5% Minimally Competent


9

Table 4. Assessments Results by Major Discipline Area

# of Projects

Assignment Mean percent of total points

earned

Cumulative Percent

Minimally competent

Competent Highly

Competent

BIS/SCPS 20 Capstone 70% 100% 85% 45%

Humanities (3 programs) 35 Course paper 60% 100% 86% 23%

Interdisciplinary 46 Capstone 77% 100% 98% 52%

Sciences (2 programs) 61 Course paper/team

project 84% 100% 83%

Social Sciences (2 programs)

56 Course paper 68% 100% 96% 33%

Commerce 48 Capstone/Team

project 88% 100% 90%

Education 20 Proposal 77% 100% 95% 70%

Engineering 36 Thesis project 64% 97% 80% 25%

Nursing 10 DMP 76% 100% 50%

Figure 3. Competency by Major Discipline Area

0%

20%

40%

60%

80%

100%

Highly Competent

Competent

Minimally competent


10

Performance by Outcome Average scores by learning outcome reinforce the finding that students’ projects reflected competency in research. The mean scores for all outcomes ranged between 3.55 and 3.86 on a scale from 1 (not competent) to 5 (highly competent), reflecting solid competency across the board. (Table 5)

Table 5. Descriptive Statistics by Learning Outcome

N Mean Score S.D.

Approach 328 3.69 0.83

Sources 293 3.65 1.01

Question 328 3.62 0.94

Data Collection 247 3.85 0.87

Data Analysis 328 3.53 0.85

Conclusion 328 3.53 0.92

Communication 328 3.67 0.85

The strong similarity of means across the seven outcomes would not be expected, however, especially in light of the variability in disciplines, assignments, and assessment processes. Rather, the nearly identical means suggest that the averaging of scores for multiple types of research projects across nine disciplines may have resulted in some regression towards the mean. The large standard deviations also suggest that each mean represents a wide range of values. Review of results by discipline reveals that some outcomes were more likely to be rated higher (e.g., Communication earned the highest mean in four of the disciplines) or lower (e.g., either Analysis or Conclusion was given the lowest mean in five of the disciplines). (Appendix E) Rank Order. To explore further, results were rank-ordered by outcome for each discipline. The outcome with the lowest average score was given a “1”; the outcome with the next lowest average score was given a “2”, etc. These rankings were averaged across disciplines by outcome to yield an average rank for each outcome. Re-ordered from highest to lowest, the results provide a general idea of relative student accomplishment by aspect of research. (Figure 4).

Figure 4. Mean Rank of Rubric Ratings by Outcome

2.44

2.56

3.50

3.67

4.11

4.29

4.33

0.00 1.00 2.00 3.00 4.00 5.00

Conclusion

Data Analysis

Sources

Research question

Communication

Data Collection

Approach

Average Rank of Accomplishment (higher=better)


11

The analysis suggests that performance was better for some learning outcomes than for others. Students were most able to approach the study of a subject or problem from a particular disciplinary or interdisciplinary point of view (e.g., “thinking like a…”), to collect pertinent data/information, and to communicate their findings clearly and effectively. Students were least able to analyze data or evidence and to draw logical and defensible conclusions. This result is not surprising as analysis and development of conclusions both require higher order thinking—the ability to weigh evidence and synthesize material. Students’ ability to manage time and to devote time to revision may also be factors. To the extent that students procrastinate or underestimate the time required to conduct the analysis, deliberate over conclusions, respond to initial criticisms and revise their work accordingly, and complete a research project, the final steps in the process—analysis and conclusions—may get short-shrift. This analysis of rankings may suggest that additional emphasis on 1) how to develop a research question, and 2) how to find and employ sources of information (e.g., library skills) would be worthwhile. Comparing higher-scoring (top third) projects with lower-scoring (bottom third) projects, lower-rated papers received significantly lower mean scores for all learning outcomes (p<.001). (Table 6) The large effect sizes suggest that, for all outcomes, the two groups differed substantially as well as in terms of statistical significance. Even among higher-rated papers, Analysis and Conclusion received the lowest mean scores, reflecting the ranked outcome findings. For the lower-rated projects, however, the lowest mean score was for Question, followed by Analysis and Conclusion. The differences between the two groups were largest for Question and Conclusion. The outcome, Question—the ability to form a research question based upon the relevant literature and/or observations,—distinguishes the two groups especially well. Again, this suggests that an emphasis on helping students select a good question for study or problem to address may have a substantial impact on students’ research experience.

Table 6. Mean Scores of Low and High Performers

Low High

High - Low Difference

Effect-Size

Approach* 2.94 4.55 1.61 0.85

Sources* 2.95 4.62 167 0.82

Question* 2.67 4.60 1.93 0.88

Collect* 3.05 4.56 1.51 0.73

Analyze* 2.73 4.46 1.73 0.86

Conclusion* 2.71 4.50 1.79 0.88

Communicate* 2.99 4.51 1.52 0.81

N 95.00 95.00

Total Score* 18.54 37.60 19.06 0.88

* Mean differences in performance are significant at p<.001.


12

These three outcomes—Question, Analysis, and Conclusion—received the greatest percentage of low ratings (1 or 2). (Table 7) This was the case regardless of whether the assessment was conducted by program faculty or by mixed groups of raters.

Indirect Assessment: Student Survey and Interview Results Data collected from surveys and structured interviews reveal that students who engage in undergraduate research are more likely to express higher levels of engagement with faculty, satisfaction with their educational experience, and confidence in their intellectual abilities. (Appendix F) Specifically, students who had conducted a significant research project were significantly (p<.05) more likely than other students to report:

More interactions with faculty, both in frequency and type of discussions

Greater engagement in their own education, such as contributing to class discussions, asking questions in class, choosing challenging courses, making class presentations, extensively revising a paper before submitting, raising their own standards for acceptable effort due to high standards of a faculty member, or helping a classmate better understand the course material

Greater satisfaction with advising, instruction, feedback, opportunities, personal contact with faculty, learning through research, and contribution of research experience to their education

Greater confidence in their ability to understand a specific field of study, conduct research, solve complex problems, think critically, write clearly and effectively, use scientific reasoning, judge value of information, and communicate to a critical audience

Greater agreement with University and departmental emphasis on teaching research methods

Greater agreement with the statement, “learning how to conduct research was an important goal for me while attending the University”

Remarkable as these survey results are, they do not serve as evidence of causation or impact. The results could also be interpreted as an indication that students who are more inclined to interact with faculty, be satisfied, and be confident in their abilities are more likely to participate in significant research.

Table 7. Percentage of Ratings by Score* for Each Learning Outcome

N 1 2 3 4 5

Approach 286 0.00% 3.50% 24.50% 39.90% 32.20%

Sources 250 2.00% 4.80% 21.20% 34.40% 37.60%

Question 286 1.00% 7.30% 26.20% 35.30% 30.10%

Collect 248 0.80% 2.80% 22.20% 37.10% 37.10%

Analysis 286 0.00% 7.30% 27.60% 40.90% 24.10%

Conclusion 286 0.70% 7.00% 27.30% 36.00% 29.00%

Communicate 286 0.00% 3.50% 26.20% 41.60% 28.70%

*The scores represent averages; as such, for example, “3” represents all scores >2, up to and including 3.


13

A better, albeit qualitative, window on impact lies with individual interviews. When asked what they had learned from doing a significant research project (other than how to collect data, etc.), students emphasized that they had learned to be persistent and resilient throughout the process of finding a mentor, collecting data, evaluating information, and dealing with setbacks. They also noted that they had learned to maintain confidence in their ability to learn in the face of intellectual challenges.


14

III. Findings

1. Based on assessment of a wide variety of research papers and projects, we can conclude that U.Va. undergraduate students are competent researchers and that many are even highly competent researchers.

2. Students are most capable at adopting and applying the methods of their chosen disciplines, including especially the collection of data or evidence. Students were also proficient in communicating their research clearly and effectively.

3. Students demonstrated the least ability to analyze data or evidence or to develop logical and defensible conclusions.

4. For two outcomes—ability to form a research question, and ability to select and use a variety of sources—additional teaching or guidance could have a substantial impact on students’ research. Both of these are rooted in the literature of the discipline.

5. Students’ proficiency appeared to vary by major discipline area, although other confounding variables, especially the nature of the assignment and the raters, likely have a substantial effect on the results.

6. The learning outcome—form a research question based on relevant literature or observation or identify and describe a problem—best distinguished the highly competent student-researchers from those who were on the lower end of competency. That is, students’ ability to define a good research question or problem to address could be considered a predictor of a successful research project.

7. The disciplines differ considerably in research methods, sources of data employed, and data collection techniques.

8. Scoring rubrics appear to provide a reasonably valid measure of students’ competence in conducting research, although the rubrics often needed to be tailored to the discipline.

9. Multiple sources of variability—differences among disciplines, assignments, rubrics, and raters—all conspire to limit the conclusions that can be drawn from this assessment.


15

IV. Recommendations

1. While overall, student research can be considered competent, even highly competent, the results also suggest that a greater emphasis on helping students identify and formulate a research question may well improve performance. To the extent that development of the research question or definition of a problem to be solved is rooted in students’ knowledge of the relevant literature, additional emphasis on enhancing students’ library research skills may be well worth the effort. 2. As the first attempt to assess undergraduate research institution-wide, this assessment encountered multiple and significant challenges that could be addressed by adjusting the assessment plan:

Research in the creative and design disciplines (fine arts and architecture) manifests differently than research in academic and applied fields, enough so that we were unable to include these disciplines in this assessment. Subsequent assessments should place high priority on developing an approach, a set of learning outcomes, and rubrics that can address student research in these disciplines.

While the committee defined four types of research (academic, applied, design, and creative), in reality many more approaches to research aligned with the separate disciplines (and majors or sub-disciplines within discipline areas) exist. Future assessments could focus on one distinct major to represent each discipline area rather than trying to accommodate the wide diversity of approaches within discipline areas.

Only faculty from a selected major should assess student research from that major. This assessment found that mixed groups of raters are more likely to disagree in their assessments, undermining the reliability of the results.

Focus on a subset of learning outcomes, perhaps those of special interest or those that are closely related (e.g., Question, Sources, and Data Collection) rather than asking raters to assess all learning outcomes. For this assessment, well over 4000 assessments were made (332 projects * 7 learning outcomes * 2 raters each). If fewer outcomes were assessed, raters would be able to devote more thought to each outcome, an important consideration given that each of the learning outcomes could easily be the subject of an in-depth assessment.


16

APPENDICES

Page

A- SERU Survey Results- Participation 17 B- Rubric for Academic Research Project 19 C- Rubric for Applied Research Project 21 D- Modified Research Project Rubrics 24 E- Results by Discipline Area 30 F- SERU 2012 Survey Results-Self-Report—Learning 65

17

APPENDIX A STUDENT EXPERIENCE IN THE RESEARCH UNIVERSITY SURVEY

2012 RESULTS RE: PARTICIPATION

Shading denotes statistically significant difference (p<.05)

Indicate the following research and creative activities that you are currently participating in or have done

Completed Sig.

Research Proj

Did not complete

Sig. Research

Proj Total

Count Percent Count Percent Count Percent

A research project, creative activity, or research

paper as part of your coursework

No 4 1.6% 48 11.9% 52 7.9%

Yes, doing now or have done 250 98.4% 355 88.1% 605 92.1%

Total 254 100.0% 403 100.0% 657 100.0%

At least one student research course No 75 29.5% 230 57.4% 305 46.6%


Total 254 100.0% 401 100.0% 655 100.0%

At least one independent study course No 143 56.3% 326 81.3% 469 71.6%


Total 254 100.0% 401 100.0% 655 100.0%

Assist faculty in research with course credit No 165 65.0% 329 81.8% 494 75.3%


Total 254 100.0% 402 100.0% 656 100.0%


18

Assist faculty in research for pay without course

credit

No 204 80.6% 367 91.8% 571 87.4%


Total 253 100.0% 400 100.0% 653 100.0%

Assist faculty in research as a volunteer without

course credit

No 206 81.1% 344 85.6% 550 83.8%


Total 254 100.0% 402 100.0% 656 100.0%

Work on creative projects under the direction of

faculty with course credit_

No 151 59.4% 310 77.3% 461 70.4%


Total 254 100.0% 401 100.0% 655 100.0%


faculty for pay without course credit

No 238 93.7% 389 97.5% 627 96.0%


Total 254 100.0% 399 100.0% 653 100.0%


faculty as a volunteer without course credit

No 213 84.5% 366 91.5% 579 88.8%


Total 252 100.0% 400 100.0% 652 100.0%

19

APPENDIX B

Rubric for Assessment of a Significant Academic Research Project

Description 5 = Highly Competent 4 3 2 1 = Not Competent Weight Score

Approach the study of a subject or problem, from a particular disciplinary or interdisciplinary point of view (e.g., “thinking like an economist”).

Approach to the study of the subject or problem reflects significant knowledge of the literature and research methodologies within the discipline or disciplines.

Approach to the study of the subject or problem reflects little or no knowledge of the literature and research methodologies within the discipline or disciplines.

Uncover and use a variety of sources, including primary sources whenever possible.

Sources are varied, high quality, pertinent, sufficient, and include primary sources where possible.

Too few sources are used, some or all are not pertinent, and available primary sources are not included.

Form a research question based upon the relevant literature and/or observations.

Research question is specific, clear, original, and thoughtful, with the potential to contribute to knowledge in the field. Literature or observations are relevant, important, and balanced, including primary sources where possible.

Research question is posed with so little clarity as to be confusing, or it is absent. Cited literature or observations lack relevance, quality, depth and/or balance.


20

Collect pertinent data/information.

Collected data are closely related to the research question. The collection methodology is clearly described, rigorous, adequate to the task, and could be replicated by another researcher.

Data are not collected or examined, or are insufficient and/or irrelevant to the research question. Data collection method is either not described, unclear, or biased.

Analyze data/information. Data analysis is clear, thorough, and appropriate to the research. Findings are presented clearly, ignoring unimportant results and highlighting the most significant ones.

Data analysis is missing, unclear, simplistic, or biased. Analysis is unrepresentatively selective or unrelated to the research question.

Draw logical and defensible conclusions.

Conclusions are present, logical, related to the research question, supported by argument and evidence

Conclusions are missing, unclear, illogical, irrelevant to the research question, or unsupported by argument or evidence.

Communicate clearly and effectively findings and conclusions.

Writing is free of grammatical, syntax and typographical errors, is well organized, and enhances the reader’s ability to understand the findings and conclusions. Word choice and tone enhance the discussion of the research.

Writing contains significant grammatical, syntax, and typographical errors and is poorly organized. Errors significantly impair the reader’s understanding of the findings and conclusions. Word choice and tone are poor.

Defend the research to a critical audience.

Presenter makes a thorough and clear presentation of the research, speaks well, and engages the audience. Presenter responds confidently and thoroughly to critical questions or feedback, enhancing the audience’s understanding of the research.

Presentation is disorganized. Presenter speaks poorly or simply reads from a text, fails to communicate key points, does not engage the audience, and/or does not respond to critical questions or feedback.


21

APPENDIX C

Rubric for Assessment of a Significant Applied Research Project

In an applied research project, a student will:

Identify a problem to be solved, or need to be addressed, based upon existing information.


Analyze data/information.




5 = Highly Competent 4 3 2 1 = Not Competent Weight Score

Identify a problem to be solved, or need to be addressed, based upon existing information.

Approach to the research question or hypotheses statement reflects a clear, practical, probabilistic, and conceptual nominal definition suitable to empirical investigation balanced with operational specification well suited to available technologies and research methods identified through scientific literature review.

11.8%


22

Uncover and use a variety of sources, including primary sources whenever possible.

Sources are varied, high quality, pertinent, sufficient, and include primary sources where possible.

Too few sources are used, some or all are not pertinent, and available primary sources are not included.

11.8%

Form a research question based upon the relevant literature and/or observations.

Research question is specific, clear, original, and thoughtful, with the potential to contribute to knowledge in the field. Literature or observations are relevant, important, and balanced, including primary sources where possible.


17.6%


Collected data are closely related to the research question. The collection methodology is clearly described, rigorous, adequate to the task, and could be replicated by another researcher.

Data are not collected or examined, or are insufficient and/or irrelevant to the research question. Data collection method is either not described, unclear, or biased.

11.8%

Analyze data/information. Data analysis is clear, thorough, and appropriate to the research. Findings are presented clearly, ignoring unimportant results and highlighting the most significant ones.

Data analysis is missing, unclear, simplistic, or biased. Analysis is unrepresentatively selective or unrelated to the research question.

11.8%


Conclusions are present, logical, related to the research question, supported by argument and evidence

Conclusions are missing, unclear, illogical, irrelevant to the research question, or unsupported by argument or evidence.

11.8%


Writing is free of grammatical, syntax and typographical errors, is well organized, and enhances the reader’s ability to understand the

Writing contains significant grammatical, syntax, and typographical errors and is poorly organized. Errors significantly

11.8%


23

findings and conclusions. Word choice and tone enhance the discussion of the research.

impair the reader’s understanding of the findings and conclusions. Word choice and tone are poor.


Presenter makes a thorough and clear presentation of the research, speaks well, and engages the audience. Presenter responds confidently and thoroughly to critical questions or feedback, enhancing the audience’s understanding of the research.

Presentation is disorganized. Presenter speaks poorly or simply reads from a text, fails to communicate key points, does not engage the audience, and/or does not respond to critical questions or feedback.

11.8%


24

APPENDIX D

RUBRICS MODIFIED BY PROGRAMS

Rubric for Assessment of a Significant Applied Research Project

ICE Final Research Group Capstone Project, Fall 2010 In an applied research project, students should do the following:

Identify a problem to be solved/addressed based upon existing information

Collect pertinent data/information

Analyze data/information

Draw logical and defensible conclusions

Communicate findings and conclusions clearly and effectively

Defend the solution to a critical audience

5=Highly competent 4 3=

Satisfactory 2 1=Not competent

1.Problem identification: extent to which the problem(s) were identified

A defined problem was formally identified and described in sufficient detail

There was no mention of the problem(s) to be addressed

2.Paradigm Application: extent to which the approach for solving the problem(s) utilized disciplinary or interdisciplinary models/paradigms

Approach to the study of the problem reflected significant knowledge of disciplinary or interdisciplinary models or paradigms

Approach to the study of the problem reflected little to no knowledge of disciplinary or interdisciplinary models or paradigms

3.Sources: extent to which sources were utilized/cited, including primary sources, whenever possible

Sources were varied, high quality, pertinent, sufficient, and included primary sources were possible

Too few sources were used, some or all were not pertinent, and available primary sources were not included


25

4.Data collection: quality of data/information collected

Collected data were closely related to the problem and the collection methodology was clearly described, rigorous, and adequate to the task.

Data were not collected or examined, or were insufficient and/or irrelevant to the problem. The data collection method was either not described, unclear, or biased

5.Analysis of data/transformation: extent to which appropriate and thorough data analysis was conducted

Data analysis was clear, thorough, and appropriate to the problem. Findings were presented clearly, ignoring unimportant results and focusing on the significant ones

Data analysis was missing, unclear, simplistic, or biased. Analysis was inappropriate and/or unrelated to the problem.

6.Conclusion development: extent to which logical and defensible conclusions were provided based on the information presented

Conclusions were presented and were logical, related to the problem in question, and supported by arguments and evidence

Conclusions were missing, unclear, illogical, irrelevant to the problem in question, and/or were unsupported by argument or evidence

7.Integrated perspective of business: extent to which the analysis and conclusions demonstrated an understanding and appreciation of the integrated nature of business

Problem definition, analysis and conclusions demonstrated an appreciation for and understanding of the integrated nature of business

Problem definition, analysis and conclusions demonstrated “silo-like” approach to different business disciplines and ignored the integrated nature of business

8.Communication of ideas: extent to which ideas were communicated clearly and effectively

Writing was free of grammatical, syntax and typographical errors, was well organized, and enhanced the reader’s ability to understand the findings and conclusions. Word choice and tone enhanced the discussion of the problem and analysis.

Writing contained significant grammatical, syntax and typographical errors and was poorly organized, Errors significantly impaired the reader’s ability to understand the findings and conclusions. Word choice and tone were poor

9.Creativity: extent to which ideas proposed were both

Ideas presented were both highly novel and potentially

Ideas presented were not at all novel and potentially useful


26

novel and potentially useful useful 10.Innovation: extent to which recommendations represented radical changes to existing practices at the sponsor’s firm

Recommendations represented distinct and meaningful changes to the existing practices at the sponsor’s firm

Recommendations represented only slight or no change to existing practices at the sponsor’s firm

Rubric for Assessment of an STS Research Project

In an STS Research Project, a student will:

Formulate and justify a research question/objective

Specify and use an appropriate approach to the research question/objective

Present and support a clear answer/describe and justify the product that meets the objective

Assess results in terms of both quality and significance

Present the results in an accessible and confidence-inspiring form

5 = Highly Competent 4 3 2 1 = Not Competent Score

Formulate and justify a research question/objective

Research question or objective is specific, clear, original, and thoughtful, with the potential to contribute to knowledge in the field. Literature or observations are relevant, important, and balanced, including primary sources where possible.



27

Specify and use an appropriate approach to the research question/objective

Approach to the research question or hypotheses statement reflects a clear, practical, probabilistic, and conceptual definition suitable to empirical investigation balanced with operational specification well suited to available technologies and research methods.

Approach to the research question or objective reflects little or no knowledge of the literature and research methodologies within the discipline or disciplines.

Present and support a clear answer- or-describe and justify the product that meets the objective.

Conclusions in the form of an answer or product are present, logically related to the research question, supported by argument and evidence

Conclusions in the form of an answer or product are missing, unclear, illogical, irrelevant to the research question, or unsupported by argument or evidence.

Assess results in terms of both quality and significance.

Analysis and results are examined in light of the research question; significant results are highlighted; methodology is clearly described, rigorous, adequate to the task, and could be replicated by another researcher.

Analysis and results are not examined, or are insufficient and/or irrelevant to the research question. Method is either not described, unclear, or biased.

Present the results in an accessible and confidence-inspiring form

Writing is free of grammatical, syntax and typographical errors, is well organized, and enhances the reader’s ability to understand the findings and conclusions. Word choice and tone enhance the discussion of the research.

Writing contains significant grammatical, syntax, and typographical errors and is poorly organized. Errors significantly impair the reader’s understanding of the findings and conclusions. Word choice and tone are poor.


28

Rubric for Assessment of an Interdisciplinary Capstone Research Paper

Capstone Course Research Paper Competency Assessment Student number ________ Rater’s initials _________ Date __________________

SCORING GUIDE:

Not Competent = 1 Minimally Competent = 2 Competent = 3 Highly Competent = 4 For each objective, write a check in the appropriate box: Objective Not

Competent Minimally Competent

Competent Highly Competent

Identified an interesting research question

Developed a cogent thesis

Gathered information from a variety of sources and demonstrated familiarity with basic research tools and methodology including citation of sources


29

Demonstrates familiarity with literature and/or foundational ideas in AAS that are directly relevant to the topic

Organized paper logically

Writing is clear and free of grammatical and spelling errors

30

APPENDIX E

RESULTS BY DISCIPLINE AREA

Assessment of Undergraduate Research Papers:

College of Arts and Sciences: Humanities

Research papers were solicited from 4000-level humanities seminars in three programs wherein the assignment required that students submit papers that reflected the full research process, from identification of the research question through analysis and conclusions. Twenty research papers were assessed according to the Rubric for an Academic Research Project. This scoring rubric consists of seven outcomes, each scored by two reviewers on a scale from 1=not competent to 5=highly competent. Assessments were conducted by faculty and selected graduate students from the humanities, social sciences, and sciences.

1. Approach the study of a subject or problem, from a particular disciplinary or

interdisciplinary point of view. (Approach)

2. Uncover and use a variety of sources, including primary sources whenever possible.

(Sources)

3. Form a research question based upon the relevant literature and/or

observations.(Question)

4. Collect pertinent data/information. (Data Collection)

5. Analyze data/information.(Analysis)

6. Draw logical and defensible conclusions. (Conclusions)

7. Communicate clearly and effectively findings and conclusions.(Communication)

A total of 35 points was possible. The mean total score was 21.10 (S.D.=4.32); that is, on average, students scored 60% of total points possible. Total scores ranged from a low of 12 points to a high of 28.

Descriptive Statistics

N Minimum Maximum Mean Std. Deviation

Total Score 20 12.00 28.00 21.10 4.32

Total Percent 20 34.29 80.00 60.29 12.36

Twenty-three percent of papers, having earned 75 percent of total points or more, would be defined as highly competent. Another sixty-three percent would be considered competent while 14 percent fell into the minimally competent range.

Institutional Assessment and Studies Core Competency Research

Assessment 2012

31

Analyzing the results by outcome, the outcome, Communication, achieved the highest ratings, averaging 3.61. The lowest scores were given for Sources and Question.

Descriptive Statistics by Outcome

N Minimum Maximum Mean Std.

Deviation

Approach 36 1.5 5.0 3.37 0.8

Sources 36 1.0 5.0 2.96 1.11

Question 36 1.0 5.0 2.94 0.85

Data collection 36 1.0 4.5 3.21 0.95

Analysis 36 1.5 4.5 3.03 0.79

Conclusions 36 1.5 4.5 3.00 0.79

Communication 36 1.5 5.0 3.61 0.77

0%

10%

20%

30%

40%

31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100%

Pe

rce

nt

of

Pa

pe

rs

Percent of Total Points Earned

Percent of Humanities Research Papers by Percent of Points Earned , n=36

23% highly competent

63% competent

14% minimally competent


Assessment 2012

32

Inter-rater reliability of ratings. Although the scoring session began with norming, the overall reliability of ratings was low (α=0.36), even when taking into account that we allow evaluators to be normed within one point of each other. For three of the seven outcomes—Question, Analyze, and Conclusions—reliability of ratings was very low. For two outcomes, however, Sources and Collection, reliability of ratings rose to moderate levels. The composition of the scoring group may contribute to these results: half of the raters were not from the humanities and they may have been able to evaluate the more basic aspects of the research (Sources and Data Collection) but may have been less able to appreciate the type of question and analysis. Further inquiry revealed one “easy grader” and one “hard grader.” The “hard grader” happened to have graded more papers than the other graders, thereby lowering the reliability scores and slightly lowering the overall assessment ratings.

Reliability of Ratings by Outcome and Overall

α Approach 0.23

Sources 0.74

Question 0.16

Data Collection 0.68

Analysis 0.15

Conclusions 0.00

Communication 0.29

Overall 0.36

Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10 percent. On average, however, 29 percent of ratings showed more than one point difference between raters; for two outcomes, Question and Conclusions, 43 percent of ratings differed by more than one point. This may suggest that raters had significant disagreement or lack of understanding about how questions are posed and conclusions drawn in research in the humanities.

3.61

3.00

3.03

3.21

2.94

2.96

3.37

1 2 3 4 5

Communicate

Conclusion

Analyze

Collect

Question

Sources

Approach

Humanities: Mean Scores by Outcome


Assessment 2012

33

Analysis of Raw Agreement by Outcome

Approach Sources Question

Data

Collection Analysis Conclusions Communication Mean

Equal

Rating

26% 43% 23% 34% 23% 20% 46% 31%

One point

difference

49% 43% 34% 49% 49% 37% 26% 41%

More than

one point

difference

26% 15% 43% 17% 29% 43% 29% 29%


Assessment 2012

34


College of Arts and Sciences: Interdisciplinary

Research papers were solicited from six 4000-level interdisciplinary capstone research seminars across five terms where the assignment required that students submit papers that reflected the full research process, from identification of the research question through analysis and conclusions. Forty-six research papers were assessed by two instructors in the program according to a scoring rubric designed by the program faculty. The research rubric consisted of six outcomes, scored on a scale from 1=not competent to 4=highly competent. Assessments were conducted by program faculty.

1. Identified an interesting research question (Question)

2. Developed a cogent thesis (Thesis)

3. Gathered information from a variety of sources and demonstrated familiarity with basic

research tools and methodology including citation of sources (Methodology)

4. Demonstrates familiarity with literature and/or foundational ideas that are directly

relevant to the topic (Literature)

5. Organized paper logically (Organization)

6. Writing is clear and free of grammatical errors (Communication)

A total of 24 points was possible. The mean total score was 18.48 (S.D.=3.67); that is, on average, students scored 77 percent of total points possible. Total scores ranged from a low of 9.5 points to a high of 24.



Total Score 46 9.50 24.00 18.48 3.67

Total Percent 46 39.58 100 77.00 15.29

Fully 98 percent of papers would be considered competent in that they achieved at least 50 percent of the total score. Fifty-two percent of papers, earning 75 percent of total points or more, would be defined as highly competent.


Assessment 2012

35

Analyzing the results by outcome, papers revealed greater strength in students’ ability to develop a research question, demonstrate familiarity with the literature, and organize the paper. Average scores were lowest for methodology and for development of a cogent thesis.



Deviation

Question 46 1.0 4.0 3.30 0.70

Thesis 46 1.0 4.0 2.95 0.82

Methodology 46 1.5 4.0 2.83 0.79

Literature 46 1.5 4.0 3.20 0.68

Organization 46 1.5 4.0 3.20 0.51

Communication 46 1.5 4.0 3.03 0.74

2.2%

15.2% 17.4%

21.7% 17.4%

26.1%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

41-50 51-60 61-70 71-80 81-90 91-100

Percent of Interdisciplinary Research Papers by Percent of Points Earned

52% Highly competent

46% competent



Assessment 2012

36

Inter-rater reliability of ratings. Inter-rater reliability of ratings was moderate to high for these papers.

Reliability of Ratings

by Outcome and Overall

α Question 0.63

Thesis 0.76

Methodology 0.55

Literature 0.71

Organization 0.53

Communication 0.67

Overall 0.80

The robust reliability scores are reflected in the analysis of raw agreement. Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10 percent. In this assessment, almost no ratings showed more than one point difference between raters. The high percentage of equal ratings suggests that raters agreed about expectations for student performance or understanding of research in this interdisciplinary major.

3.0

3.2

3.2

2.8

2.9

3.3

1 2 3 4

Communication

Organization

Literature

Methodology

Thesis

Question

Interdisciplinary: Mean Scores by Outcome


Assessment 2012

37

Analysis of Raw Agreement by Outcome and Overall

Question Thesis Method Literature Organization Communication Mean

Equal Rating 78% 83% 81% 87% 81% 80% 82%

One point

difference

22% 17% 17% 13% 17% 20% 18%

More than

one point

difference

0% 0% 2% 0% 2% 0% 0%


Assessment 2012

38

Assessment of Undergraduate Research Projects

College of Arts and Sciences: Science

Research projects were assessed from two programs. The preponderance of reports was from a year-long 4000-level course designed to promote and support team laboratory research projects and presentations. The assignment required that students conduct laboratory research that reflected the full research process, from identification of the research question through analysis, conclusions, and presentation. This was the only research in the sample wherein an oral defense of the research was required and assessed. A small number of projects were solicited from senior seminars in a science program. Fifty-two students’ projects were assessed according to the Rubric for an Academic Research Project. This scoring rubric consists of eight outcomes, each scored by one reviewer on a scale from 1=not competent to 5=highly competent.




(Sources)







8. Defend the project to a critical audience. (Defense)

A total of 40 points was possible. The mean total score was 33.72 (S.D.=4.75); that is, on average, students scored 84 percent of total points possible. Total scores ranged from a low of 21 points to a high of 40.


N Minimum Maximum Mean

Std.

Deviation

Total Score 52 21.00 40.00 33.72 4.75

Total Percent 52 52.50 100.00 84.30 11.87

Eight-three percent of papers earned 75 percent of total points or more and would be defined as highly competent. Another 17 percent were judged to be competent.


Assessment 2012

39

Analyzing the results by outcome, all mean scores fell within a range from a low of 4.10 to a high of 4.46. The highest score was for Data Collection. Other outcomes also received high ratings: Approach, Sources, Analysis, and Communication. The lowest mean score was for the defense of the project.



Deviation

Approach 52 3.0 5.0 4.21 0.72

Sources 52 2.0 5.0 4.21 0.78

Question 52 2.0 5.0 4.17 0.76

Data Collection 52 3.0 5.0 4.46 0.70

Analysis 52 2.0 5.0 4.21 0.78

Conclusions 52 3.0 5.0 4.13 0.66

Communication 52 3.0 5.0 4.22 0.71

Defense 43 2.0 5.0 4.10 0.72

10%

2%

25%

35% 29%

0%

10%

20%

30%

40%

50%

60%

51-60 61-70 71-80 81-90 91-100

Pe

rce

nt

of

Pa

pe

rs

Percent of Possible Points Earned

Percent of Research Projects by Percent of Points Earned


17% competent


Assessment 2012

40

Inter-rater reliability of ratings. For most of the assessments, projects were assessed by a single rater. For the other projects in this group, the sample size is too small to evaluate inter-rater reliability.

4.10

4.22

4.13

4.21

4.46

4.17

4.21

4.21

1 2 3 4 5

Defend

Communicate

Conclusions

Analyze

Collect

Question

Sources

Approach

Mean Scores by Outcome


Assessment 2012

41


College of Arts and Sciences: Social Science

Research papers were solicited from 4000-level social science seminars where the assignment required that students submit papers that reflected the full research process, from identification of the research question through analysis and conclusions. Forty-eight research papers were assessed according to the Rubric for an Academic Research Project. This scoring rubric consists of seven criteria, each scored by two reviewers on a scale from 1=not competent to 5=highly competent.




(Sources)



4. Collect pertinent data/information. (Collect)

5. Analyze data/information.(Analyze)


7. Communicate clearly and effectively findings and conclusions.(Communicate)

A total of 35 points was possible. The mean total score was 23.75 (S.D.=4.16); that is, on average, students scored 68% of total points possible. Total scores ranged from a low of 14.5 points to a high of 32.5.



Total Score 48 14.50 32.50 23.75 4.16

Total Percent 48 41.43 92.86 67.86 11.88

With a cut-off of 50% of points earned, all papers would be considered at least competent. Fully one-third of papers earned 75% of total points or more and would be defined as highly competent.


Assessment 2012

42

Analyzing the results by criterion, two criteria, all mean scores fell within a close range from a low of 3.23 (Conclusions) to 3.47 (Communicate).



Std.

Deviation

Approach 48 1.50 4.50 3.45 0.64

Sources 48 2.00 5.00 3.41 0.77

Question 48 2.00 5.00 3.44 0.72

Collect 48 2.00 4.50 3.42 0.70

Analyze 48 1.50 4.50 3.34 0.72

Conclusions 48 1.50 5.00 3.23 0.81

Communicate 48 2.00 5.00 3.47 0.71

4%

25%

31%

19% 19%

2%

0%

10%

20%

30%

40%

50%

60%

41-50% 51-60% 61-70% 71-80% 81-90% 91-100%

Percentage of Possible Points Earned

Percent of Social Science Papers by Percent of Points Earned


63% competent



Assessment 2012

43

Inter-rater reliability of ratings. Although the scoring session began with norming, the overall reliability of ratings was very low (α=0.17), even when taking into account that we allow evaluators to be normed within one point of each other. For two of the seven criteria—Approach and Conclusions, reliability was 0.0. For only one criterion, Sources, did reliability of ratings rise to moderate levels. Further inquiry revealed one “easy grader” and one “hard grader.” The “hard grader” happened to have graded more papers than the other graders, thereby lowering the reliability scores and slightly lowering the overall assessment ratings.



α Approach 0.00

Sources 0.44

Question 0.12

Collect 0.16

Analyze 0.18

Conclusions 0.00

Communicate 0.26

Overall 0.17

Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10%. On average, however, 25% of ratings showed more than one point difference between raters and for only one criterion (Sources) did the percent fall lower than 20%. This suggests that raters had significant disagreement about expectations for student performance or understanding of research in the humanities. Raters particularly struggled with the criterion, Conclusions, where 35% of the pairs of ratings differed by more than one point.

3.47

3.23

3.34

3.42

3.44

3.41

3.45

0 1 2 3 4 5

Communicate

Conclusions

Analyze

Collect

Question

Sources

Approach

Social Science: Mean Scores by Criterion


Assessment 2012

44

Analysis of Raw Agreement by Criterion and Overall

Approa

ch Sources

Questio

n Collect Analyze

Conclusion

s

Communicat

e Mean

Equal

Rating

23% 40% 33% 23% 25% 25% 33% 29%

One point

difference

52% 44% 40% 52% 52% 40% 42% 46%

More than

one point

difference

25% 16% 27% 25% 23% 35% 25% 25%


Assessment 2012

45

Assessment of Undergraduate Research Capstone Projects: Commerce

Research projects were solicited from the capstone research seminars where the assignment required that students conduct applied research in teams and submit papers that reflected the full research process, from identification of the research question through analysis and conclusions. Forty-eight ICE team research papers were assessed according to a scoring rubric designed by the program faculty. Derived from the Rubric for an Applied Research Project, the rubric consists of ten outcomes, each scored by multiple reviewers on a scale from 1=not competent to 5=highly competent. Sixteen papers were assessed by two reviewers, sixteen by three reviewers, and sixteen by five reviewers.

1. Problem identification: extent to which the problem(s) were identified Uncover and use

a variety of sources, including primary sources whenever possible.

2. Model/Paradigm Application: extent to which the approach for solving the problem(s)

utilized disciplinary or interdisciplinary models/paradigms

3. Sources: extent to which sources were utilized/cited, including primary sources,

whenever possible

4. Data collection: quality of data/information collected

5. Analysis of data/transformation: extent to which appropriate and thorough data

analysis was conducted

6. Conclusion: extent to which logical and defensible conclusions were provided based on

the information presented

7. Integrated perspective of business: extent to which the analysis and conclusions

demonstrated an understanding and appreciation of the integrated nature of business

8. Communication: extent to which ideas were communicated clearly and effectively

9. Creativity: extent to which ideas proposed were both novel and potentially useful

10. Innovation: extent to which recommendations represented radical changes to existing

practices at the sponsor’s firm

A total of 50 points was possible. The mean total score was 43.80 (S.D.=2.90); that is, on average, students scored 88 percent of total points possible. Total scores ranged from a low of 32 (64%) points to a high of 50 (100%).



Std. Deviation

Total Score 48 32.00 50.00 43.80 2.90

Total Percent 48 64.00 100.00 87.60 6.71


Assessment 2012

46

Ninety percent of papers earned 75 percent of total points or more and would be defined as highly competent. Ten percent were judged to be competent, and 15 percent to be minimally competent.

Analyzing the results by outcome, all mean scores fell within a range from a low of 4.10 to a high of 4.67. Among the outcomes shared by the other disciplines, the lowest mean scores were for Communication (4.19) and the highest for Model/Paradigm (Approach) (4.64).



Std.

Deviation Problem (Question) 48 3.8 5.0 4.64 0.39

Model/Paradigm (Approach) 48 3.3 5.0 4.56 0.36

Sources 48 3.3 5.0 4.55 0.43

Data collection 48 3.3 5.0 4.30 0.45

Analysis 48 3.3 5.0 4.30 0.42

Conclusion 48 3.6 5.0 4.40 0.35

Business Perspective 48 4.0 5.0 4.67 0.33

Communication 48 2.3 5.0 4.19 0.5

Creativity 48 3.3 5.0 4.14 0.42

Innovation 48 3.0 4.8 4.10 0.43

2.08%

12.50%

43.75% 41.67%

0%

10%

20%

30%

40%

50%

60%

70%

61-70% 71-80% 81-90% 91-100%

Pe

rce

nt

of

Pa

pe

rs

Percent of Possible Points

Percent of SCC Research Papers

by Percent of Possible Points Earned


10% competent


Assessment 2012

47

Inter-rater reliability of ratings. Results were analyzed according to the three sets of sixteen papers each. Set #1 had two reviewers, Set #2 had five reviewers, and Set #3 had three reviewers. Reliability of ratings was generally low to moderate.

Reliability of Ratings by Outcome and Overall for 3 Sets of Projects

Total

Set #1

(n=16)

2 reviewers

Set #2

(n=16)

5 reviewers

Set #3

(n=16)

3 reviewers

N α α α

Problem ID 48 0.559 0.338 0.750

Model/Application 48 ---* 0.231 0.726

Sources 48 ---* 0.582 0.624

Data Collection 48 0.392 0.108 0.685

Analysis 48 0.072 0.463 0.620

Conclusions 48 --* 0.063 0.208

Business Perspective 48 --* 0.536 0.350

Communicate 48 0.426 0.559 0.703 Creativity 48 0.804 0.530 --*

Innovation 48 0.549 0.341 0.510

Overall 48 0.605 0.717 0.833

*Ratings showed too little variance to assess reliability.

Ideally, we would like to see the percentage of ratings that differ by more than one point to be around 10 percent. On average, however, 17 percent of ratings showed more than one point difference between raters. The greatest disagreements were found for Problem Identification (21%) and Creativity (19%).

4.10

4.14

4.19

4.67

4.40

4.30

4.30

4.55

4.56

4.64

1 2 3 4 5

Innovation

Creativity

Communication

Business Perspective

Conclusion

Analysis

Data Collection

Sources

Model/Paradigm

Problem



Assessment 2012

48

Raw Agreement by Outcome and Overall

Problem Model Sources Collect Analyze Conclusions Integrate Communicate Creativity Innovation

Equal Rating 54% 29% 29% 29% 35% 10% 35% 27% 10% 29%

One point difference

25% 58% 60% 58% 54% 73% 48% 58% 71% 54%

More than

one point

difference

21% 13% 10% 13% 10% 17% 17% 15% 19% 17%


Assessment 2012

49

Assessment of Undergraduate Capstone Research Projects: SCPS: Bachelor of Interdisciplinary Studies

Research projects were solicited from the capstone seminars where the assignment required

that students undertake a “sustained critical evaluation of a thesis that demonstrates the

student's deep understanding of salient research and viewpoints in the relevant research

community.” Students’ projects were to assert an important research question, demonstrate

mastery of methodology, data collection and analysis, pose clear and compelling arguments for

the thesis, be well organized, and clearly communicate the project in writing. Research topics

and approaches were highly diverse, ranging from historical analysis of a crusade against comic

books to a project on standardized testing for kindergarten students to an analysis of the impact

of the Marion Dogma on proto-feminism.

Twenty research projects were assessed according to the Rubric for an Academic Research Project. This scoring rubric consists of seven outcomes, each scored by two reviewers on a scale from 1=not competent to 5=highly competent. The assessment was conducted by program faculty.




(Sources)










Total Score 20 14.50 34.00 24.75 6.12

Total Percent 20 41.43 97.14 70.71 17.49


Assessment 2012

50

Forty-five percent of papers earned 75 percent of total points or more and would be defined as highly competent. Forty percent were judged to be competent, and 15 percent to be minimally competent.

Analyzing the results by outcome, all mean scores fell within a range from a low of 3.30 to a high of 3.93. The lowest mean scores were for Analysis (3.30) and for Conclusions (3.35).



Std.

Deviation

Approach 20 2.50 5.00 3.60 0.75

Sources 20 2.50 5.00 3.93 0.86

Question 20 1.50 5.00 3.58 1.10

Data Collection 20 2.00 5.00 3.60 0.99

Analysis 20 1.50 5.00 3.30 1.06

Conclusions 20 1.00 5.00 3.35 1.10

Communication 20 2.00 5.00 3.40 0.99

15% 15%

20%

10%

30%

10%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

41-50% 51-60% 61-70% 71-80% 81-90% 91-100%

Pe

rce

nt

of

Pa

pe

rs


Percent of BIS Capstone Projects

by Percent of Points Earned


40% competent 15% minimally competent


Assessment 2012

51

Inter-rater reliability of ratings. The overall reliability of ratings was moderate (α=0.46), probably attributable to the fact that the reviewers were all program faculty. For the outcome, Approach, however, reliability was 0.0, suggesting that the reviewers did not share the same expectations for how the students should approach the student of a problem. This may be an expected outcome in that, in this Bachelor of Interdisciplinary Studies students may take highly diverse approaches to their research, including complex interdisciplinary approaches.

Reliability of Ratings by Outcome and Overall

α Approach 0.00

Sources 0.49

Question 0.63

Data Collection 0.49

Analysis 0.49

Conclusions 0.46

Communication 0.68

Overall 0.46

Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10 percent. On average, however, 26 percent of ratings showed more than one point difference between raters. While the greatest disagreement is found for Approach (45% with more than one point difference), all the other criteria showed percentages of 20-30

3.40

3.35

3.30

3.60

3.58

3.93

3.60

1.00 2.00 3.00 4.00 5.00

Communicate

Conclusions

Analyze

Collect

Question

Sources

Approach



Assessment 2012

52

percent. While this may suggest that raters had significant disagreement about expectations for student performance, only one-quarter of the 20 papers accounted for nearly half of the differences in scores.



Data Collection Analysis Conclusions Communication Mean

Equal Rating 25% 45% 35% 25% 15% 15% 40% 29%

One point

difference 30% 35% 45% 50% 60% 55% 40% 45%

More than

one point

difference

45% 20% 20% 25% 25% 30% 20% 26%


Assessment 2012

53

Assessment of Undergraduate Research Proposals: Education

Research proposals were solicited from a Kinesiology course where the assignment required that students create research proposals that reflected the full research process, from identification of the research question through plans for analysis and development of conclusions. As these were not reports of completed research projects, the papers described how data would be collected and analyzed. The assessment took into account that the papers were proposals, not reports of completed research. Twenty papers were assessed according to the Rubric for an Academic Research Project. This scoring rubric consists of seven outcomes, each scored by two reviewers on a scale from 1=not competent to 5=highly competent. Assessment was conducted by program faculty.




(Sources)










Std.

Deviation

Total Score 20 16.50 32.00 27.00 4.55

Total Percent 20 47.14 91.43 77.14 12.99

Seventy percent of papers earned 75 percent of total points or more and would be defined as

highly competent. Another 25 percent were judged to be competent and 5 percent to be

minimally competent.


Assessment 2012

54

Analyzing the results by outcome, all mean scores fell within a range from a low of 3.38 to a

high of 4.13. The highest scores were given for Approach, Data Collection, and Communication

and the lowest mean scores were for Analysis (3.38).



Approach 20 2.50 5.00 4.13 0.63

Sources 20 1.00 5.00 3.75 1.23

Question 20 2.00 5.00 3.80 0.89

Data Collection 20 2.50 5.00 4.10 0.70

Analysis 20 2.00 4.00 3.38 0.60

Conclusions 20 2.00 5.00 3.78 0.88

Communication 20 2.00 5.00 4.08 0.83

5% 10%

5%

40%

30%

10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

41-50 51-60 61-70 71-80 81-90 91-100

Pe

rce

nt

of

Pa

pe

rs

Percent of Possible Points

Percent of Research Proposals by Percent of Points Earned

70% highly competent 25% competent



Assessment 2012

55

Inter-rater reliability of ratings. The overall reliability of ratings was moderate (α=0.53), probably attributable to the fact that the reviewers were all program faculty. For the outcome, Analysis, however, the reliability score was 0.07, suggesting that the reviewers did not share the same expectations for how well the students should have been able to plan for the data analysis.



α Approach 0.54

Sources 0.89

Question 0.46

Collect 0.52

Analyze 0.07

Conclusions 0.54

Communicate 0.69

Overall 0.53

Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10 percent. On average, however, 19 percent of ratings showed more than one point difference between raters. It appears that raters had particular difficulty agreeing on the Question and Conclusions outcomes.

4.08

3.78

3.38

4.10

3.80

3.75

4.13

1 2 3 4 5

Communicate

Conclusions

Analyze

Collect

Question

Sources

Approach



Assessment 2012

56

Analysis of Raw Agreement by Criterion and Overall

Approach Sources Question Collect Analyze Conclusions Communicate Mean

Equal

Rating

40% 50% 30% 40% 40% 35% 60% 42%

One point

difference

55% 40% 35% 50% 35% 35% 25% 39%

More than

one point

difference

5% 10% 35% 10% 25% 30% 15% 19%


Assessment 2012

57

Assessment of Undergraduate Research Papers: SEAS

Fourth-year SEAS students complete a thesis portfolio that consists of a technical report focused on an engineering problem and a Department of Engineering and Society research paper that “provides evidence of your ability to inform yourself about issues related to the broader environmental, social, and ethical contexts of engineering practice or technology, employ conceptual resources (including STS perspectives) and critical thinking to explore these issues, to form a reliable judgment concerning them, and to write about the issue in a persuasive and professional way.” Both research projects are completed during the students’ fourth year. Although the technical report has long been a requirement for graduation, the addition of the DES research paper that focuses on the contexts of engineering is relatively new (and still evolving), as is the portfolio structure for the thesis. Thirty-six DES research projects were assessed according to a scoring rubric designed by the program faculty. Derived from the Rubric for an Academic Research Project, the rubric consists of five criteria, each scored by two reviewers on a scale from 1=not competent to 5=highly competent. Assessment was conducted by faculty from SEAS, CLAS, and SON.

1. Formulate and justify a research question/objective (Question)

2. Specify and use an appropriate approach to the research question/objective (Approach)

3. Present and support a clear answer- or-describe and justify the product that meets the

objective. (Conclusions)

4. Assess results in terms of both quality and significance. (Analysis)

5. Present the results in an accessible and confidence-inspiring form (Communication)

A total of 25 points was possible. The mean total score was 15.92 (S.D.=3.91); that is, on average, students scored 64 percent of total points possible.



Std.

Deviation

Total Score 36 6.50 23.50 15.92 3.91

Total Percent 36 26.00 94.00 63.67 15.64

Twenty-five percent of papers earned 75 percent of total points or more and would be defined as highly competent. Just over half projects earned scores that would put them in the competent range. Not all projects could be considered competent, however, as 17 percent were deemed minimally competent and 3 percent (1 paper) not competent.

http://www.sts.virginia.edu/common/pdf/UTM2012-2013.pdf


Assessment 2012

58

Analyzing the results by outcome, there was a wide range between minimum and maximum for all outcomes, with the mean scores settling within a narrow range between 3.03 and 3.33. The outcome Approach received the lowest mean score, perhaps a reflection of the relative newness of this assignment, that is, asking engineering students to evaluate the environmental, social, and ethical aspects of engineering. The highest ratings were given to two outcomes, Conclusions and Communication.



Std.

Deviation

Question 36 1.00 5.00 3.19 0.91

Approach 36 1.50 4.50 3.03 0.81

Conclusions 36 1.00 5.00 3.25 0.91

Analysis 36 1.50 5.00 3.11 0.87

Communication 36 1.50 5.00 3.33 0.84

0%

10%

20%

30%

40%

50%

Pe

rce

nt

of

Pa

pe

rs


Percent of STS Research Projects by Percent of Possible Points Earned


55% competent 17% minimally competent

3% not competent


Assessment 2012

59

Inter-rater reliability of ratings. Although the scoring session began with norming, the overall reliability of ratings was low (α=0.20), even when taking into account that we allow evaluators to be normed within one point of each other. Further inquiry revealed one non-SEAS rater as rating significantly differently than others, and within the SEAS raters, there was one "hard" rater and one "easy" rater. This contributed significantly to the low reliability. It also contributed to the mean scores hovering around “3”, the middle score.



α Question 0.00

Approach 0.13

Conclusions 0.27

Analysis 0.40

Communication 0.18

Overall 0.20

Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10 percent. Overall, however, raters differed by more than one point 38 percent of the time, suggesting that raters disagreed about expectations, standards, application of the rubric, etc. However, it should be observed that 14 percent of the 36 projects accounted for 27% of the differences. Raters particularly struggled with the outcome re: Question where over half of the pairs of ratings differed by more than one point.

3.33

3.11

3.25

3.03

3.19

1 2 3 4 5

Communication

Analysis

Conclusions

Approach

Question

SEAS: Mean Scores by Outcome


Assessment 2012

60

Analysis of Raw Agreement by Outcome and Overall

Question Approach Conclusions Analysis Communication Mean

Equal

Rating

17% 28% 28% 31% 17% 24%

One point

difference

25% 39% 42% 39% 44% 38%

More than

one point

difference

55% 33% 30% 31% 39% 38%


Assessment 2012

61

Assessment of Undergraduate Distinguish Majors Projects:

Nursing

Fourth-year students in the Distinguished Majors program “pursue a substantive project of their own that they would not have the opportunity to develop as part of their regular program of study. The application process requires students to define a detailed research project and to obtain agreement of a faculty member to guide the research. The final product is a manuscript ready to submit for publication as well as an oral presentation.”6 Ten DMP research projects were assessed according to the Rubric for an Academic Research Project. This scoring rubric consists of seven outcomes, each scored by two reviewers on a scale from 1=not competent to 5=highly competent. Assessments were conducted by program faculty.




(Sources)







Total scores were computed and reviewers’ scores were averaged for each outcome and for the total score. Analysis was conducted using the mean scores. A total of 35 points was possible. The mean total score was 26.60 (S.D.=3.95); that is, on

average, students scored 76 percent of total points possible.



Std.

Deviation

Total Score 10 21.00 34.00 26.60 3.95

Total Percent 10 60.00 97.14 76.00 11.28

All papers could be considered competent. Fifty percent of papers earned 75 percent of total points or more and would be defined as highly competent.

6 http://records.ureg.virginia.edu/content.php?catoid=31&navoid=1206

http://records.ureg.virginia.edu/content.php?catoid=31&navoid=1206


Assessment 2012

62

Analyzing the results by outcome, the highest ratings were given to three outcomes: Approach, Sources, and Communication, although the mean scores for the seven outcomes varied little (range: 3.60-3.95).



Std.

Deviation

Approach 10 3.00 5.00 3.95 0.50

Sources 10 3.00 4.50 3.90 0.57

Question 10 2.50 5.00 3.60 0.88

Data Collection 10 3.00 5.00 3.75 0.63

Analysis 10 3.00 5.00 3.75 0.82

Conclusions 10 3.00 4.50 3.70 0.67

Communication 10 3.00 5.00 3.95 0.83

10%

20%

40%

10%

20%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

51-60% 61-70% 71-80% 81-90% 91-100%

Percent of Nursing DMP Projects by Percent of Points Earned

50% Highly competent 50% Competent


Assessment 2012

63

Inter-rater reliability of ratings. Reliability of ratings, both by outcome and overall, was moderate.



α Approach 0.54

Sources 0.69

Question 0.55

Collect 0.66

Analyze 0.74

Conclusions 0.53

Communicate 0.79

Overall 0.64

Ideally, we would like to see the average percentage of ratings that differ by more than one point to be around 10 percent. For two outcomes, Sources and Data Collection, disagreement was substantially higher than for the other outcomes. Three of the papers account for this result as the reviewers disagreed by two or even three points on each paper for these two outcomes. It is not clear what it was about these three papers that elicited such disagreement between reviewers regarding use of sources and data collection.

3.95

3.70

3.75

3.75

3.60

3.90

3.95

0.00 1.00 2.00 3.00 4.00 5.00

Communication

Conclusions

Analysis

Data Collection

Question

Sources

Approach

Nursing: Mean Scores by Outcome


Assessment 2012

64



Data

Collection Analysis Conclusions Communication Mean

Equal

Rating

80% 20% 70% 30% 50% 50% 70% 53%

One point

difference

10% 40% 10% 30% 50% 40% 20% 29%

More than

one point

difference

10% 40% 20% 30% 0% 10% 10% 18%


Assessment 2012

65

APPENDIX F

STUDENT EXPERIENCE IN THE RESEARCH UNIVERSITY (SERU) SURVEY

2012 RESULTS FOR SELECTED QUESTIONS

Shading denotes significant difference (p<.05)

During this academic year how frequently have you done the following:

Scale from 1 (Never) to 6 (Very often)

Completed Sig.

Research Proj

Did not complete

Sig. Research Proj Total

Mean N Mean N Mean N

Contributed to a class discussion 4.70 254 4.38 404 4.50 658

Brought up ideas or concepts from different

courses during class discussions

4.21 253 3.74 404 3.92 657

Asked an insightful question in class 4.01 252 3.65 400 3.79 652

Found a course so interesting that you did more

work than was required

3.66 254 3.14 401 3.35 655

Chosen challenging courses, when possible, even

though you might lower your GPA by doing so

4.31 254 3.93 404 4.08 658

Made a class presentation 4.35 254 3.67 403 3.93 657

Had a class in which the professor knew or

learned your name

5.05 254 4.65 404 4.81 658

Taken a small research-oriented seminar with

faculty

3.12 252 2.07 403 2.48 655

Communicated with a faculty member by e-mail

or in person

5.16 251 4.67 403 4.86 654

Talked with the instructor outside of class about

issues and concepts derived from a course

4.09 253 3.48 402 3.72 655

Interacted with faculty during lecture class

sessions

4.00 253 3.58 402 3.74 655

Worked with a faculty member on an activity

other than coursework (e.g., student

organization, campus committee, cultural

activity)

2.75 252 2.12 401 2.37 653

Turned in a course assignment late 1.68 254 1.66 403 1.67 657

Gone to class without completing assigned

reading

3.35 254 3.26 403 3.29 657

Gone to class unprepared 2.90 253 2.82 403 2.85 656


Assessment 2012

66

Skipped class 2.23 252 2.29 400 2.27 652

Raised your standard for acceptable effort due to

the high standards of a faculty member

3.89 254 3.66 402 3.75 656

Extensively revised a paper at least once before

submitting it to be graded

4.18 254 3.77 402 3.93 656

Sought academic help from instructor or tutor

when needed

3.53 251 3.23 399 3.34 650

Worked on class projects or studied as a group

with other classmates outside of class

4.26 254 3.93 403 4.06 657

Helped a classmate better understand the course

material when studying together

3.96 254 3.59 402 3.74 656

During this academic year how frequently have you done the following: Scale from 1 (Never) to 6 (Very often)

Completed Sig. Research Proj

Did not complete Sig. Research

Proj

Total

Turned in a course assignment late 1.68

Gone to class without completing assigned 3.35 reading

Gone to class unprepared 2.90

Skipped class 2.23

1.66

3.26

2.82

2.29

3.66

3.77

3.23

Raised your standard for acceptable effort due to the high standards of a faculty member

Extensively revised a paper at least once before submitting it to be graded

Sought academic help from instructor or tutor when needed

3.89

4.18

3.53

During this academic year how frequently have you done the following: Scale from 1 (Never) to 6 (Very often)

Completed Sig. Research Proj

Did not complete Sig. Research

Proj

Total



Assessment 2012

67

Worked on class projects or studied as a group with other classmates outside of class

Helped a classmate better understand the course material when studying together

4.26

3.96

254

254

3.93

3.59

403

402

4.06

3.74

657

656

Please rate your current level of proficiency in the following areas

Scale from 1 (Very poor) to 6 (Excellent)

Completed Sig.

Research Proj

Did not complete

Sig. Research Proj Total


Analytical and critical thinking skills - Current

ability level

5.13 250.0 4.88 399.0 4.98 649.0

Ability to be clear and effective when writing -

Current ability level

4.86 251.0 4.69 399.0 4.76 650.0

Ability to read and comprehend academic

material - Current ability level

5.00 247.0 4.81 393.0 4.88 640.0

Foreign language skills - Current ability level 3.50 251.0 3.62 398.0 3.57 649.0

Understanding of a specific field of study -


5.11 247.0 4.90 396.0 4.98 643.0

Quantitative (mathematical and statistical) skills

- Current ability level

4.12 247.0 4.06 399.0 4.08 646.0

Ability to speak clearly and effectively in

English - Current ability level

5.35 249.0 5.17 397.0 5.24 646.0

Ability to understand international perspectives

(economic, political, social, cultural) - Current

ability level

4.82 252.0 4.66 399.0 4.72 651.0

Leadership skills - Current ability level 4.81 252.0 4.54 400.0 4.65 652.0

Computer skills - Current ability level 4.79 251.0 4.60 398.0 4.67 649.0

Internet skills - Current ability level 5.16 245.0 5.04 399.0 5.08 644.0

Library research skills - Current ability level 4.59 246.0 4.31 399.0 4.42 645.0

Other research skills - Current ability level 4.82 245.0 4.40 395.0 4.56 640.0

Ability to prepare and make a presentation -


4.97 246.0 4.66 399.0 4.78 645.0

Interpersonal (social) skills - Current ability level 5.00 246.0 4.87 399.0 4.92 645.0

Ability to appreciate, tolerate and understand

racial and ethnic diversity - Current ability level

5.08 250.0 5.06 399.0 5.07 649.0

Ability to appreciate the fine arts (e.g., painting,

music, drama, dance) - Current ability level

4.76 250.0 4.70 399.0 4.72 649.0

Ability to appreciate cultural and global diversity 5.00 247.0 4.94 396.0 4.96 643.0


Assessment 2012

68

- Current ability level

Understanding the importance of personal social

responsibility - Current ability level

5.09 247.0 5.06 397.0 5.07 644.0

Self-awareness and understanding - Current

ability level

5.16 250.0 5.15 398.0 5.16 648.0

Documents

UNDERGRADUATE RESEARCH AT THE UNIVERSITY OF VIRGINIA · 2017-02-17 · Given this understanding, the assessment of undergraduate research competency would have to provide for differences