C1

Project Description CCLI Phase 2: Building Support Structures for Full Adoption of the Affinity Research Group Model

1. Introduction

1.1 Overview of Project Undergraduate research is a well-known approach to integrate knowledge and provide practice of the skills critical to business, industry, and government, in particular, refinement of cognitive and interpersonal skills, enhancement of personal growth, and inculcation of intellectual and management habits. While a common practice is to recruit and involve the most visibly successful students, this limits the number of promising students who can benefit from research experiences. To extend the research experience to a broader range of students, particularly students from underrepresented groups, The University of Texas at El Paso (UTEP) successfully developed and implemented the Affinity Research Group (ARG) model (Kephart et al., 2008), which provides students with opportunities to learn, use, and integrate the knowledge and skills that are required for research with those required for collaborative work. The model creates an integrated research environment in which a group of diverse students and faculty contribute to the research effort. Through the ARG model, faculty mentors create and sustain a cooperative environment that explicitly develops skills to make students successful in research, academe, and the workforce. As a result, students and faculty, in particular those from underrepresented groups, can reach higher levels of productivity and achievement. Over a five-year period (1998-2003), enrollment of Hispanic citizens in UTEP’s Computer Science graduate programs increased from 18% to approximately 40% of the total graduate enrollment. This percentage has remained steady. Evaluation of the ARG model (Gates et al., 1999c; Della Piana & Bernat, 1999; Kephart & Villa, 2008) showed that students felt that they were able to transfer ARG techniques they had learned, such as how to resolve conflict, give constructive criticism, brainstorm solutions to problems, ask questions, and communicate with team members, to group situations outside of their research groups. An unexpected outcome of the ARG model has been the fostering of a commitment by members to help other students succeed in computer science and research.

With funding from NSF CCLI and the IEEE, the investigators have published an ARG handbook (www.computer.org/arg; edited copy for review only: http://cahsi.cs.utep.edu/Portals/0/ARGbook.pdf) and delivered numerous workshops across the country on the model. Most of the workshops have focused on awareness, and some provide practice of ARG activities. Interviews with faculty members post-workshop indicate that adopting ARG is challenging and that more support is needed early in the adoption process.

UTEP proposes a Phase II project that will focus on widespread adoption of the ARG model in STEM fields through creation of a mentoring and collaboration structure centered on ARG Hubs. An ARG Hub, similar to the one at UTEP, is a collection of three or more faculty who 1) use the ARG model to cultivate a climate where research skills and other ARG attributes are deliberately and intentionally practiced; 2) provide a support structure for faculty adopting the ARG model; and 3) document effective activities and practices. The proposal will support four geographically and culturally diverse coalitions:

• Alliance for the Advancement of African-American Researchers in Computing (A4RC) (Lead Mentor: Maureen Biggers): The NSF-sponsored A4RC matches computer science research faculty from Historically Black Colleges and Universities with colleagues from Research-1 institutions. HBCU Master’s and undergraduate students are added to each “research pod”; the collaborative teams work throughout the year, participating in a spring research methods course, on-going research that connects to a 10-week summer internship as a team in the R1 lab, and post-internship project-related work with the goal of publishing and presenting posters at professional conferences. While teamwork is emphasized throughout the experience, formal training in high performing teams has not yet been a part of the A4RC model; and they are interested in adopting ARG model as they sustain and expand their initiatives with young African-American computing students. Current participating institutions include North Carolina A&T, Jackson State University, Norfolk State University, Bennett College, Indiana University, University of Colorado, Georgia Institute of Technology, and Virginia Tech. Several other HBCU’s have expressed an interest in participating in this project, and a proposal is being written to expand participation to include these interested computing departments. Several

C2

A4RC faculty and students participated in a short ARG workshop that introduced them to this approach and they have asked for more formal training in how to implement it effectively.

• Computing Alliance of Hispanic-Serving Institutions (CAHSI) (Lead Mentor: Nayda Santiago with Ann Gates): CAHSI has adopted ARG as an effective practice for developing students’ technical, professional, and research skills. CAHSI, funded by the NSF Broadening Participation in Computing program, is a consortium of seven institutions (California State University-Dominguez Hills, Florida International University, New Mexico State University, Texas A&M-Corpus Christi, University of Houston-Downtown, University of Puerto Rico-Mayaguez, and UTEP) that is focused on the recruitment, retention, and advancement of Hispanic students in computing. CAHSI adopters have requested faculty development workshops and mentoring support to guide their adoption of ARG. The proposed project will develop the training needed to allow current adopters to gain depth of knowledge and expertise. As a result, it will allow them to work with other faculty at their home institutions to disseminate the model more broadly.

• University of Nebraska Omaha (Lead Mentor: Dana Richter-Egger): An interdisciplinary team of math and science faculty at The University of Nebraska at Omaha (UNO) is adopting ARG for use in undergraduate research programs. The team is also interested in applying the ARG model more broadly to support the development of students’ professional and leadership skills through UNO’s Math-Science Learning Center (MSLC), which will lead the AGR adoption at UNO. The MSLC is part of the College of Arts and Sciences and is a resource both for faculty and students focused on the academic success of UNO’s math and science students. The University of Nebraska at Omaha is a metropolitan, primarily undergraduate institution (~12,000 undergraduate students) and is the largest institution of higher education in the Omaha metropolitan area: population ~ 825,000.

• Virginia Tech University, Graduate School (Lead Mentor: Manuel A. Pérez-Quiñones): Virginia Tech is adopting ARG as a way to mentor undergraduates from STEM fields interested in pursuing graduate school. The Office of Graduate Student Diversity will create affinity groups that include students of similar ethnic backgrounds but cross discipline boundaries. In the groups senior students, incoming graduate students and faculty mentors will work together, fostering student connectedness and reinforcing skills development.

Over the funding period, mentors in disciplines inside and outside computing at each Hub will develop the knowledge, skills, and experience to support and sustain ARG research groups. This endeavor will result in a broader support network of ARG adopters and advocates who will disseminate the model, contribute to the ongoing development of ARG best practices, and contribute to the body of knowledge concerning the conditions and factors that enable ARGs and their members to thrive.

1.2 Motivation With globalization and the increasing competitiveness of other countries in science, technology, engineering, and mathematics (STEM), the United States is in danger of losing its leadership in these areas unless it mobilizes its efforts to increase participation in STEM. Efforts such as the American Competitiveness Initiative and the report by the President’s Council of Advisors on Science and Technology (PCAST, 2007) further support the need to invest in development of our youth to ensure our country’s leadership in STEM and economic growth. The increase in the number of jobs requiring STEM skills continues to outpace job development in the U.S. labor force as a whole. Effective recruitment and education strategies must be developed to reverse the dwindling numbers of students entering STEM.

Undergraduate research is a well-known approach for engaging students in their discipline and moving them into advanced degree programs. The success of NSF’s Research Experiences for Undergraduates (REU) and other similar programs is a testament to the value of research experiences. Undergraduate research provides the means for students to integrate knowledge and practice skills critical for the workforce (Hakim, 2000; Kurz-Milcke, 2004; Laursen, 2006). In particular, employers seek graduates with project management, communication, and team skills (McGee, 2007), as well as those adept at problem-solving, conflict management, leadership, critical thinking, professionalism, and ethics (Browning, 1995; NCWIT, 2004; Kaplan, 2007; McConnell, 2003). The past decade has seen a dramatic increase in the recognition of the power of collaborative product development practices (Sawyer, 2007). Today’s employees must be able to work in culturally and professionally diverse teams.

C3

The common practice in undergraduate research is to recruit and involve the most visibly successful students. But this approach fails to encourage a large number of promising students -- students who have the capability to succeed in research but may lack one or more of the following: role models, the confidence to seek involvement in research, or awareness of opportunities or benefits of being involved in research (often the result of being a first-generation college student). The National Academy of Engineering’s booklet The Engineer of 2020 states: “We aspire to an engineering profession that will effectively recruit, nurture, and welcome underrepresented groups to its ranks” (NAE, 2004 p. 50). Indeed, to have impact on graduate degree production, it is critical that U.S. women and under-represented groups be recruited into STEM graduate programs. To extend the undergraduate research experience to a broader range of students, UTEP successfully developed and implemented the Affinity Research Group (ARG) model (Gates, Delgado et al. 1997; Gates, Teller et al., 1999a; Bernat, Teller et al., 2000a) with NSF funding. The ARG model was further developed in the NSF CCLI-EMD project, The Affinity Research Group Model: Developing Students Beyond Academe (Gates et al., 2008; Roach and Gates, 2008; Kephart et al., 2008). The project was successful in developing learning materials, in particular a published faculty-centered handbook described later in this proposal, and in disseminating the model to seven institutions through workshops.

1.3 Basis of Proposed Work The proposed work will build on knowledge gained from past ARG efforts on structuring and sustaining productive research environments. We will also build on the experiences of faculty members who have attempted to adopt ARG practices.

In addition to a significant increase in the number of U.S. citizens (in particular those from underserved groups) who entered graduate studies in computing at UTEP, a retrospective study of ARG alumni (Kephart & Villa, 2008) found how student participation in ARG influenced their personal and professional development of skills and disciplinary knowledge. Using ethnographic interviews, evaluators gathered descriptions of alumni experiences in ARGs to gauge the long-term effects of these experiences. ARG alumni readily described specific aspects of the ARG model, such as paper and presentation critiques, through which they developed technical and social skills that they continue to use in the workplace and that they believe have contributed significantly to their professional mobility and success. For example, one alumna, who is employed as a technical writer in a Fortune 50 company, reports that she is well prepared for the rigors of a fast-paced, results-driven work environment. She said that participation in an ARG prepared her to “not be afraid to ask questions” and also taught her “how to communicate, how to write.” In addition to technical and social skills, physical location emerged as a key element in creating a supportive environment for ARG members. That is, creating workspaces where ARG members are physically close to each other allows them to interact easily with each other, promotes positive interdependence, and supports a flat hierarchy.

To ground the findings, the study related components of the ARG model to the science of learning (Bransford & Committee, 2000), in particular socio-cultural learning theory (Vygotsky & Kozulin, 1986). Under the ARG model, research groups are formed around shared goals (technical/research, as well as cooperative/social) that newer members come to appreciate more deeply through participation in the groups’ activities under the guidance of more experienced members. The theory of situated cognition/learning (Lave & Wenger, 1991) offers a framework for elucidating how learning and development occur through participation in personally and socially meaningful practices. In an ARG, students are engaged in a supportive and affirming learning environment where the goals and processes of research are explicit. As a result, participants are gradually able to make sense of what they are learning; turn these new experiences into abstract ideas; and, ultimately, transfer what they are learning into new situations. ARGs embody what Lave and Wenger (1991) refer to as communities of practice, a form of social organization in which a common underlying purpose motivates the group’s activities and facilitates the organization of these activities into a body of established practices. The ARGs' continuous and deliberate focus on the development of research, professional, and team skills in a cooperative environment provides students with an optimal situation for developing skills and abilities that are highly valued in academia and industry. As evidence of this, the comments of the alumni strongly suggested that they were conscious of having benefited from an unusual learning experience, one that few of their

C4

counterparts at their university and later in their workplaces have been fortunate enough to have had. The interviewees articulated stories of their time in the ARG that illustrated how they gradually became aware of the goals and purposes of the research group and, through intense mentoring from ARG peers and faculty, developed the skills that enabled them to confidently and competently complete research tasks, present the findings of their research within and outside the group, and make significant contributions to the group’s publications. In turn, as their skills and confidence developed, they became mentors to newer ARG members, thus continuing the learning/mentoring cycle.

While the investigators have given numerous workshops to disseminate the ARG model and the participants have expressed enthusiasm about adopting the model, it is clear that workshops are not enough. Common remarks are: “I see the benefits of ARG, I’m excited about the model, but I don’t know where to start. I need to know more.” “I’m implementing ARG activities, but how do I know if I’m doing it right?” These remarks, feedback from CAHSI adopters, and discussions with the ARG Advisory Team (see Section 3.2) indicate the need for intensive faculty development activities to allow prospective ARG faculty mentors to practice and critically reflect on the nuances of the model.

1.4 Goals, Intellectual Merit, and Broader Impact The overarching goal of the project is to create a network of ARG adopters who have the depth of knowledge needed to support widespread ARG adoption. The goals of the proposed project are to: Goal 1 Develop faculty expertise by creating a self-sustaining support structure through which ARG

adopters can learn, practice, and critically reflect on ARG core components. Goal 2 Improve the ARG model by reinforcing (addressing) the factors that enable (hinder) the adoption

of the ARG model at geographically diverse institutions and across a variety of disciplines. Goal 3 Determine the impact and success of ARG in those settings.

This proposal requests support for the following efforts in support of the goals: (1) Establish an ARG Professional Development Program. We will create and deliver a professional development program that will provide the participants with the knowledge, tools, skills, capability, and confidence to implement an ARG. (2) Implement a Support Structure: Faculty members adopting an ARG need a strong support structure, and additional faculty and student learning materials must be created, particularly outside of the computing field. (3) Assess and Evaluate Adoption: To facilitate the adoption of the ARG model, we must better understand the factors that lead to success. (4) Assess and Evaluate Impact: Evaluators will collect demographic and experiential data from current and former ARG participants to understand the impact of adoption at the Hubs and determine the level of increase in the number of students who proceed to graduate studies. The expected outcomes of the project are given in Fig. 2.

Intellectual Merit. Unlike models that focus on a one-on-one mentor relationship, the ARG model creates an integrated research environment in which diverse students and faculty contribute to the research effort. It provides a framework and pedagogy that enable faculty to create and sustain a cooperative environment that explicitly develops skills to make students successful in research, academe, and the workforce. As a result, students and faculty can reach higher levels of productivity and achievement. This is critical as the U.S. seeks to educate our youth in STEM areas and advanced studies to promote innovation and remain competitive. The proposed project will establish ARG Hubs at geographically and culturally diverse institutions and establish a broad support network of ARG adopters and advocates who can disseminate and support adoption of the ARG model. ARG advocates from a wide range of research groups and institutions will extend the materials presented in the ARG handbook to include a greater variety of successful practices. The development of an on-line repository of materials will provide domain-specific materials for researchers outside of computing and engineering. The close interaction with the adopting institutions and research groups will lead to the identification of barriers to the adoption of the ARG model and the development of strategies to overcome them. Dissemination of the ARG model and the deliberate development of students’ research, professional, and communication skills will lead to significant improvements in the quality of research and education and to a higher level of competence among a larger number of graduates.

Broader Impacts. Students involved in research develop skills beyond those typically learned in the classroom, in particular, refinement of cognitive and interpersonal skills, enhancement of personal growth,

C5

and inculcation of intellectual and management habits. Additionally, involvement in research improves student persistence. While the undergraduate research experience has often been restricted to the “best and the brightest” students, the ARG model aims to include students who historically have not been involved in research. The proposed ARG Professional Development programs will train faculty, especially young faculty, in ARG techniques, strategies, and philosophies that result in retaining and advancing students, especially those from underserved groups, in STEM fields. Based on past results, we anticipate an increase in the number of students, particularly those from underrepresented groups, who are involved in research and continue to graduate school. The implementation of the ARG model will improve the ability of students to transition to the workplace and will address the national need for a diverse and highly qualified STEM workforce. In addition, the ARG model provides lightweight process improvement approaches that can be easily incorporated in a variety of small group settings.

2. Background: The Affinity Research Group Model In 1995, the ARG model was conceived at UTEP with the goal of involving CS and electrical and computer engineering undergraduate students in research to improve recruitment, retention, and persistence of students, particularly female students and students from underrepresented populations. UTEP is an urban university whose ethnic composition mirrors that of El Paso with an 80% Hispanic population. It is a commuter school, and a significant portion of the undergraduate population is the first generation in the family to attend college. In 1995, few CS students were on campus except to attend classes, and few continued to graduate school. The introduction of ARGs changed the culture of the CS department, increasing student-faculty interaction outside the classroom. A larger network of students formed study groups in their college careers, and students stayed on campus longer.

An ARG is a team of faculty mentors and students who work together cooperatively to accomplish a research task. The ARG model joins two foundational ideas: that interaction among students and faculty outside the classroom increases the likelihood of students persisting to graduation (Astin, 1985; Rodriguez, 1994; Tinto, 1993), and that cooperative learning techniques maximize student learning and efficacy (Johnson et al, 1989b). The model promotes the inclusion of a variety of students with different levels of experience, expertise, interests, skills, and cultural and educational backgrounds. The ARG model embraces this diversity and exposes students to experiences that facilitate the development and transfer of knowledge and skills among members of the group. By integrating best practices from a variety of sources in industry, research, and education, the model provides a framework for the deliberate development and practice of skills needed by effective employees and by leaders in research, academia, and industry.

The three core components of the ARG model are: • The definition of a group’s core ideology - affinity research groups adopt a core ideology consisting of

two essential components: core values and a sense of purpose that goes beyond reaching research project goals.

• Active fostering of student connectedness, i.e., building connections among members of the group and to members of the broader discipline or profession – such connectedness is fostered through, among other means, an annual orientation and assignment of tasks. Assigned tasks and other structured activities create positive interdependence and create opportunities for promotive interaction.

• Application of deliberate management practices that reinforce skills development and promote establishment of cooperative teams - the management of an affinity research group includes project definition, scheduling of regular meetings, structured activities, and leadership in group processing for continuous quality improvement.

A key element of the ARG model is the use of the cooperative learning paradigm (Johnson & Johnson, 1989a; Johnson et al., 1990; Johnson et al., 1991; Johnson et al., 1992; Johnson et al., 1994). Cooperative groups create higher quality products, achieve mastery of or competence in a task, develop a social network, and have increased self-esteem. Structured cooperative learning techniques are integrated into the routine functioning of the group. The mere formation of a group, as in traditional research groups, does not ensure that it will function cooperatively. As Johnson and colleagues note (Johnson et al., 1990, p. 4), “Cooperation is working together to accomplish shared goals. Within

C6

cooperative activities, individuals seek outcomes that are beneficial to themselves and beneficial to all other group members.”

In an ARG, group members work together to maximize their own and each other’s productivity and achievement. The ARG model ensures that structured cooperative learning activities are part of the group’s routine functioning. Because teaching and practicing professional skills are part of the research group activities, for example, students are able to learn skills from their groups and transfer them to other environments.

Five basic elements must be present for the group to truly function cooperatively: positive interdependence, face-to-face promotive interaction, individual and group accountability, professional skills development, and group processing. The ARG model incorporates all five by structuring them into weekly activities and the group’s day-to-day functioning. Positive interdependence is the situation where each team member’s success depends on the success of the team as a whole. Face-to-face promotive interaction occurs when students are situated so they can easily and comfortably talk to each other and actively seek participation from each other. The explicit goal of this sharing process is for members to help one another succeed and, therefore, help the group reach its goals. Individual and group accountability is needed to ensure that individuals participate fully. Each person must be responsible for contributing her or his fair share to the group. Likewise, the group as a whole is responsible for the group’s smooth functioning and for delivering the required work. Professional skills are the communication and inter-personal skills that are needed to work with people in a business environment and that facilitate working relationships. In an ARG, professional skills are explicitly taught and practiced in activities focused on one or more technical topics such as critiquing a presentation, practicing active listening and asking questions. Group processing is the critical evaluation of the performance of the group. It consists of individuals assessing the quality of their contributions to the group as well as the group’s assessment of its recent performance.

3. Prior Results This section presents prior NSF-funded projects that involve the ARG model. The projects have established the foundation for the proposed work. Section 5 presents a comparison of the prior projects and the proposed work.

3.1 NSF EIA-9522207: Ann Q. Gates (PI) with Andrew Bernat, Patricia Teller, and Sergio Cabrera

a) Grant Information: NSF EIA-9522207, $1,200,000, September 1995-August 2000. b) Title: Building Affinity Groups to Enable & Encourage Student Success in Computing c) Summary: The project goal was to address retention and participation of traditionally

underrepresented groups in the computing areas by defining a framework that involved undergraduate and graduate students in research and outreach programs. The objectives were: to provide an environment that supports persistence; to define activities that develop students’ research, technical, group, and communication skills; to document and disseminate the infrastructure for creating and managing effective research groups that involve students with a wide range of abilities and experiences; and to develop a formative evaluation instrument.

d) Summary of results: All objectives were met. The major deliverables were development of a draft ARG handbook and dissemination of the model through publications, presentations, and workshops. Indicators of success of the model include an increase in student research participation, publications and invited presentations on the model; adoption of the model in other departments and universities; and grant awards. Indicators of success in student development and persistence include student conference attendance, publications, presentations, outreach involvement, retention, continuation to graduate school, awards, and scholarships. Evaluation of the ARG model indicated that: 1. students gained the skills and abilities to work effectively with others; 2. they fostered a commitment to helping each other succeed individually and collectively; and 3. ARG students attended graduate school in greater numbers than non-ARG students.

e) Selected publications and workshops: (Gates et al., 1997; Gates et al., 1997; Gates et al., 1998; Gates et al., 1999a; Gates 1998b; Gates et al. 1999b; Gates et al., 1999c; Della Piana & Bernat,

C7

1999; Bernat et al., 2000a; Bernat et al., 2000b; Teller & Gates, 2000; Gates & Della Piana, 2001; Teller & Gates, 2002).

3.2 DUE-0443061: A. Gates with S. Roach and E. Villa a) Grant Information: NSF DUE-0443061, $304,811, 08/01/05-05/31/09 b) Title: The Affinity Research Group Model: Developing Students Beyond Academe c) Summary: This project supported the effort to disseminate the model to other institutions and other fields and to encourage students to persist in STEM fields, participate in undergraduate research, and prepare to enter the workforce. The mission of the project was twofold: to document and distribute ARG best practices that develop research, technical, communication, and team competencies of diverse students and to maximize students’ potential to become more effective scientists and engineers through the use of the ARG model. The goals of the project were to present workshops to a diverse set of institutions, expand and refine the ARG handbook to address a broader audience, identify and promote activities that support best practices, and publish and distribute the ARG model. The project actively engaged an Advisory Team that provided excellent guidance in the execution of the grant, in particular in the development of the materials and workshop. The members of the Advisory Team were: Dr. Jill Singer (Professor of Earth Science and Director of the Office of Undergraduate Research, State University College at Buffalo; Past President, the Council on Undergraduate Research), Dr. Karl Smith (Professor of Engineering Education and Civil Engineering, Purdue University and University of Minnesota, respectively; nationally recognized for his work in engineering education and cooperative learning and teamwork), Dr. Ramon Lopez (Professor of Physics, University of Texas-Arlington; leader in science education); and Wendy Carmody (Director of HR, Hewlett-Packard Co.). As the ARG project leaders documented practices and developed dissemination and transfer products, the ARG Advisory Team provided suggestions and critiques. At the August 2007 meeting, the ARG Advisory Team strongly recommended that the project leaders seek a second round of CCLI funding.

d) Summary of Results: The project has resulted in two major deliverables: workshops geared toward faculty, students, and a mixed group of faculty and students; and a published ARG handbook and website. Over the past four years, almost 100 faculty members from science, computing, mathematics, and engineering programs at 19 different institutions throughout the U.S. and Puerto Rico have attended ten ARG workshops.

Project evaluation has shown that the attendees gained an interest in and understanding of the ARG key components through active engagement and reflection on a selected ARG activity. The workshops have evolved based on feedback from the evaluation, making the workshops more interactive. In addition, the evaluation team has continued to investigate ARG effectiveness (building on results from the MII project described in the previous subsection) by implementing a qualitative design (see Bogdan & Biklen,1998; Creswell, 1998) in which the evaluators conducted several one-on-one and focus group interviews with ARG alumni with the objective of gathering alumni descriptions of their experiences in the research group and gauging the long-term effects of these experiences. See Section 1.3 for further discussion. e) Publications and dissemination efforts: The project focused on publication of the ARG handbook entitled The Affinity Research Group Model: Creating and Maintaining Effective Research Teams (Gates, Roach et al., 2008). Based on the recommendation of a professional copy editor who provided invaluable guidance in its presentation, the handbook was written specifically for faculty mentors. The investigators received an IEEE Seed Initiative grant to support its publication by IEEE Computer Society Press and publicity through the IEEE Computer Society website (IEEE, 2007). In addition, the investigators have created a website to support dissemination of the model (ARG, 2007). Finally, the investigators have documented the refinement of the model in a Summer 2008 publication of the Council on Undergraduate Research (CUR) Quarterly (Kephart, Villa et al., 2008) and described how the model is applied to a software engineering course in a chapter in Software Engineering: Effecting Teaching and Learning Approaches and Practices (Roach & Gates, 2008). As described previously, Kephart and Villa (2008) published a retrospective study of ARG alumni.

4. Proposed Work As described in Section 3.2, there is evidence of widespread and growing interest in the ARG model; however, faculty mentors have expressed a need for additional support. While the previous CCLI project

C8

has accomplished the goal of disseminating the model, there are elements of the ARG model that, despite being presented in a workshop, require practice and reflection in order for the participants to become effective ARG mentors. The proposed project has three main goals: Goal 1 Create a self-sustaining support structure through which ARG adopters can learn, practice, and

critically reflect on ARG core components. Goal 2 Improve the ARG model by reinforcing (addressing) the factors that enable (hinder) the adoption

of the ARG model at geographically diverse institutions and across a variety of disciplines. Goal 3 Determine the impact and success of ARG in those settings. The expected outcomes (enumerated in Fig. 2) include expanded adoption of ARG, a well-defined support structure for faculty who engage diverse students in research, and a significant increase over time in graduate school enrollment by participants. The proposed project goals align with the recommendations of the current project Advisory Team: dissemination outside of the discipline of computer science and extension of the workshop. Goal 1: Create a self-sustaining support structure through which ARG adopters can learn, practice, and critically reflect on ARG core components.

Using feedback from the workshop evaluations, the ARG Advisory Team recommended that the length of the ARG workshop be extended in order to provide guided practice. It is important that adopters realize that ARG practices can be implemented in a variety of ways and that integrating them into a research group requires understanding and practice of the basic elements of cooperative teams and other ARG elements. In addition, ARG Hubs will be established to provide a critical mass of ARG practitioners who can share experiences and strategies through scheduled meetings and create and refine the materials and activities used in ARG processes. Together, the practitioners will form local and extended communities that contribute to the body of knowledge and practices making ARG groups effective and that serve as loci of dissemination of the ARG model.

Effort 1.1 Establish an ARG Professional Development Program

Because it takes time and reflective practice to gain expertise, the proposed work will incorporate an ARG Professional Development Program that includes a series of workshops lasting 3-5 days. The proposed summer program will provide faculty with an opportunity to gain the knowledge, skills, attitudes, and beliefs of an ARG faculty mentor. We recognize that all participants are at varying levels of understanding and expertise. As a result, the program will be designed using active and interactive tools that will allow all participants to develop their ARG skill sets in a way that is tailored to their individual levels of understanding (Carrier, 1984; Schèon, 1983). The faculty mentors will each be asked to bring one or two of their research undergraduate students in order to provide them the experience of implementing the ARG model through structured activities. Having students participate in the program creates an environment that simulates the ARG research group and provides an authentic experience for the faculty mentors, enabling them to connect theory with practice. Mentors will lead students in activities in which they practice technical and professional skills in the ARG setting. Both mentors and students will critically reflect on these activities.

The ARG Professional Development Program is focused on moving adopters from awareness to proficiency in implementing ARG components through guided instruction. The Development Program will be offered at two levels. In Year 1 of the grant, all participants will be at Level 1; in subsequent years, new adopters will be at Level 1 and continuing adopters will be at Level 2.

Level 1: Fundamentals. ARG adopters will learn about the essential elements of an ARG, including how research groups can operate as cooperative teams and plan activities that structure the essential elements of ARG. Workshop attendees will receive feedback from leaders on how ARG was practiced during the week and engage in reflection on essential elements of ARG. The objective of the Fundamentals program is to move adopters from an ad hoc model of research groups to a model with a reproducible process and systematic evaluation.

The expected outcomes for the Level 1 Development Program are that faculty participants will be able to

C9

• plan an ARG activity that structures cooperative elements; • explain the differences between an ARG and traditional research groups; • synthesize the feedback from the facilitator on their practice of ARG activities and improve based on

the feedback; • use the ARG Handbook as a resource; • create a realistic plan for implementing ARG components throughout the year. Level 2: Advanced, ARG adopters will learn how to assess the research group, engage in deeper

discussions of ARG core purpose and philosophy, videotape and critique adopters’ practice of ARG over the year, provide observation and feedback to each other on the practice of ARG, reflect on and discuss challenges to adoption, and make recommendations for improvement. The objective of the “Advanced” program is to move adopters to a model that deliberately practices cooperation and engages in and documents process improvement.

The expected outcomes for the Level 2 Development Program are that faculty participants will be able to • define their research group’s core purpose; • provide constructive critique of another participant’s ARG skills; • adjust an ARG activity to a new situation; • develop mechanisms to self-assess levels of performance in implementing an ARG; • develop cooperative skills during student–faculty interaction in their research groups. Each year, at least 20 and up to 30 faculty members will attend the Development (summer) program.

Effort 1.2 Implement a Support Structure.

The ARG Hubs will broaden the support structure for ARG adopters by creating a community that will enculturate members into ARG values and practices. Hub faculty members will develop new materials, arising from their own ARG practices, that will contribute diverse experiences and thought to the current repository.

Effort 1.2 includes regularly scheduled telephone conference calls and further development of ARG resources. Because our experience demonstrates that attending one ARG workshop is not enough to translate ARG precepts into action and launch an ARG, Hubs and their faculty members need more support to become established. Two approaches will be incorporated to provide support to each Hub. The first will provide asynchronous support through the ARG website, developing a community in which adopters can post questions, concerns, or observations on different threads of an electronic bulletin board and search for postings on a particular issue (allowing an open exchange among the ARG adopter community). The second approach provides synchronous support through regularly scheduled teleconferences in which ARG Hub participants will discuss the progress and problems of adoption. Building an ARG community will help the investigators improve the summer program and develop faculty resources.

This effort focuses on building resources to support adopters and making them freely and easily available on the Internet through the website. For the first two years of the project, printed resources will also be subsidized. But by the last year, a nominal fee will be assessed to cover expenses and to address the issue of sustainability. Faculty will benefit from having resources and guidance for developing student researchers and addressing research group challenges. Students will have resources and guidance for developing technical and professional skills and conducting research. The investigators will be able to expand the knowledge base of effective practices, allowing the model to extend to more diverse disciplines and communities.

Goal 2: Improve the ARG model by reinforcing (addressing) the factors that enable (hinder) the adoption of the ARG model at geographically diverse institutions and across a variety of disciplines. Under the current project, the evaluation effort has involved collection of demographic data about ARG participants and has utilized a survey to assess ARG participants’ and workshop attendees’ perceptions of their experiences with the model. These tools were developed to validate the ARG model, assess its effectiveness, and assess the effectiveness of the dissemination workshops in developing participants’

C10

understanding of the precepts of the model. The proposed work will include an evaluation plan that incorporates new approaches for understanding and evaluating the efficacy of the model, including ethnographic observations of ARGs-in-action and interviews with current ARG members and ARG alumni. The evaluation team has piloted the interviews, and the findings have corroborated and augmented earlier findings about the model’s efficacy for retention and promotion of students and development of their professional skills (Kephart & Villa, 2008).

Effort 2.1 Evaluate the effectiveness of the ARG Professional Development program

Effort 2.1 focuses on evaluating the Professional Development program, using first-hand observation as well as analysis of recordings made during the summer program. The evaluation team will assess the activities of the program and discern whether or not they were successful in exposing mentors and students to the terms and concepts of cooperative learning at a basic level so that participants can supply basic definitions of them. During and after the summer program, the team will also assess how well mentors are able to apply the concepts learned during the program in familiar and new situations. In the aftermath of the program, such work will take place concurrently with the work of Effort 2.3.

Effort 2.2 Investigate the role of specific individual and contextual factors.

Effort 2.2 will investigate mentors’ attitude toward cooperative learning and about the institutional support for undergraduate research needed for or impeding full implementation of the model. To accomplish this effort, it will be necessary to probe and further articulate the defining features and characteristics of the model that constitute its effectiveness. The following research questions will guide evaluation of the proposed project: • Which aspects of the model are essential to its implementation in order to achieve the goals of

enhancing student retention, increasing undergraduates’ enrollment in graduate school, and developing participants’ research and professional skills and abilities?

• What types of activities and forms of interaction are essential for preparing mentors and students to be able to implement the model?

• What types of attitudes, predispositions, and ideologies toward cooperative learning and leadership do successful mentors display?

• What types of institutional factors would support implementation, and conversely, what types of institutional obstacles might impede implementation?

In order to answer these questions, the evaluation team plans a three-pronged approach that will include assessing the impact of the proposed week-long summer program, monitoring the provision of ongoing technical support for mentors, and conducting annual site visits and/or analyzing video-recorded research group activities at institutions where ARGs are being implemented. Each of these approaches is described more fully below. • Observation and video-recording of the summer program. The evaluation team will observe and take

field notes during the program (Spradley,1980). They will also transcribe and analyze video-recordings of workshop activities in order to investigate to what extent the activities and interactions of the institute prepare participants for implementation of their own ARGs. Through observation of participants during the workshops and post-hoc analyses of video transcripts, the evaluators will gauge participants’ understanding of the model and mentors’ attitudes toward cooperative organization and their abilities to implement ARG activities.

• Ongoing support for mentors. The investigators will provide ongoing support to ARG mentors through teleconference meetings and the ARG website. The evaluators will audio-record the teleconference conversations and analyze transcripts to gauge the ongoing implementation processes. Evaluators will look for evidence in these transcripts, as well as in the materials and queries posted on the ARG website, of mentors’ ideologies concerning the mentor-student relationship, their attitudes toward cooperation, their successes in implementing ARG-related activities within their research groups, and the challenges they perceive in implementation. The evaluators will also attempt to discern the underlying forces that support and impede successful implementation.

• Site visits. The evaluation team will conduct actual or virtual site visits annually. Virtual site visits are video recordings made by Hub lead mentors of an orientation session or another ARG-related activity, such as a presentation critique session or a meeting that targets a specific social or teaming skill. The evaluation team will analyze the recordings to address the aforementioned research

C11

questions. The team will also interview mentors and student members of the ARG to further probe issues, challenges and successes of implementation.

The evaluation plan will assist adopting faculty in acquiring and evaluating their skills in mentoring students. The analysis of all data collected as part of the evaluation plan will deepen understanding of the challenges and approaches to full adoption of the ARG model. This evaluation will enable the project team to refine the summer program and will inform further development of materials to support adoption of the ARG model.

Goal 3: Determine the impact and success of ARG in diverse settings. The two efforts associated with Goal 3 are:

Effort 3.1: Collect demographic data. This effort will determine the impact of ARG by collecting data on the number of ARG students who continue to and are successful in advanced studies.

Effort 3.2: Collect experiential data. This effort focuses on continued documentation of the impact of the ARG experience on past ARG participants. The following research questions guide the data collection:

• In former ARG participants’ descriptions of their experiences in the ARG as well as professionally since graduation, what evidence is there that changes in the graduates have occurred?

• Have experiences in ARG enhanced the participants’ abilities to cope with professional challenges?

5. Comparison of Proposed Project to Previous Work

Figure 1: Comparison of funded projects.

Fig. 1 compares the projects that have funded development of the ARG model. The model was developed initially with funding through CISE-MII as a mechanism for creating a student culture that supports the retention and advancement of students into graduate studies. The current CCLI project will end in May 2009. That project focused on development of workshops and refinement of the handbook to support dissemination of the ARG model. Based on evaluation of the workshop and feedback from an Advisory Team, the investigators realized the necessity to provide a stronger support system for adopters and to extend the workshops to a summer program in order to provide adopters with guided practice.

The outcomes of the proposed project include refinement and creation of materials to enhance the research experience for undergraduates; in particular it will add activities that faculty mentors can use to develop student skills, and it will add new templates that can be used by student researchers. The proposed project will also develop faculty expertise not only in the ARG model, but in the cooperative

C12

learning paradigm, which it is at the core of the model. As a result, the techniques learned in the summer workshop will transfer to the participants’ teaching strategies.

The refinement of the model and supporting resources, as well as the proposed changes in the dissemination efforts and supporting materials/resources, have been informed by formative evaluation and learning theory. It is the belief of the project leaders that the proposed work will build a solid base for continued, self-sustaining national dissemination.

6. Project Management Management of the project is a collaborative effort with each investigator having oversight responsibility for a key component of the project as noted in Figure 2. The Leadership Team is composed of Dr. Ann Gates, Dr. Steve Roach, Dr. Kerrie Kephart, and Ms. Elsa Villa. As the PI, Ann Gates will manage the overall project and ensure that the goals, objectives, and milestones are met.

6.1 Qualifications of Investigators Ann Gates was one of the original investigators in the development of the ARG model and continues to play a key role in its implementation and refinement. She has published, with others, more than twenty papers on the model. In addition, she has presented the model in panels and given workshops. Gates is a member of the IEEE-Computer Society (IEEE-CS) Board of Governors (2004-2006); the IEEE-CS, Educational Activities Board (1997-present); the National Academy of Engineering’s Committee on Engineering Education (2002-2004); the steering committee for the Frontiers in Education Conference (2000-2002; 2003-2005); and the IEEE-CS Certified Software Development Professional Certification Committee. She is a founding member of the Computing Alliance for Hispanic-Serving Institutions as well as the Academic Alliance for the National Center for Women in Information Technology. In addition, she is a program evaluator for the Computing Accreditation Committee of ABET. Responsibilities. Gates will oversee the project and coordinate the efforts of this project with other synergistic efforts including adoption of the ARG model by CAHSI institutions, IEEE Computer Society efforts to promote the handbook, and organizations that focus on development of young faculty (e.g., SACNAS, CRA-W, and others). Gates will work with Roach and Villa to deliver the summer program. Steve Roach has been using cooperative learning and the ARG model in his courses and research since 1999. In 2002 and 2003, he chaired the IEEE CCSE Sub-Committee on Advanced Software Engineering Curricula. The CCSE is an international organization developing models of undergraduate and graduate software engineering programs. In 2003, he chaired the panel session, “The Art of Getting Students to Practice Team Skills,” at the 33rd ASEE/IEEE Frontiers in Education Conference (with E. Villa, J. Sullivan, R. Upchurch, and K. Smith). He is an IEEE-CS Certified Software Development Professional and a program evaluator for the Computing Accreditation Committee of ABET. Responsibilities. Roach will lead the development of resources to support the summer program and maintain the website. In addition, Roach will work with Gates and Villa to deliver the summer program. Kerrie Kephart conducts research in the area of social interaction, student learning, and the formation of disciplinary learning communities. She has been a co-evaluator on the current CCLI-EMD project and presented preliminary results of the evaluation at the May 2007 meeting of National Center for Women & Information Technology (NCWIT) in Boulder, CO. As a co-principal investigator on another NSF CCLI-funded project, Cultivating Authentic Discourse for the 2020 Engineer, she has been investigating the effects of inquiry-based teaching methods on engineering students’ development of deep conceptual understanding and the formation of their identities as engineers. She was also co-author of a presentation entitled, “Talking the IE Talk and Walking the IE Walk: Fostering IE Identity with Discourse Analytic Methods” (with A. Pennathur), at the 2007 Industrial Engineering Research Conference. Kephart led the writing team for the 2008 publication of "The Affinity Research Group Model: Creating and Maintaining Dynamic, Productive, and Inclusive Research Groups" in the Council on Undergraduate Research (CUR) Quarterly (Kephart, Villa et al., 2008). In addition, she conducted a qualitative study with E. Villa, "Demonstrating Sustainable Success: Using Ethnographic Interviews to Document the Impact of the Affinity Research Group Model," which was presented at the 2008 FIE Conference (Kephart & Villa, 2008).

C13

Responsibilities. Kephart will lead the evaluation and assessment of the project. She will work with Villa to implement the evaluation plan and visit adopting institutions. Elsa Villa served as Director of the Engineering Programs Office in the UTEP College of Engineering from 1994 to 2005. In that capacity, she worked closely with faculty members in developing and implementing recruitment and retention strategies. She is currently a doctoral student in curriculum and instruction at New Mexico State University and is a multi-year lecturer for the UTEP Department of Teacher Education where she is a member of the Division of Science, Mathematics, and Technology. Her research interests include professional learning communities in education and inquiry learning in teacher preparation. Ms. Villa is a certified cooperative learning trainer through the University of Minnesota Cooperative Learning Center. Responsibilities. Villa will lead the effort to build and interact with the ARG communities and to provide support on the ARG model via teleconferencing. She will assist Kephart in the program evaluation and will work closely with the other investigators in developing the summer program materials, refining existing documents, and designing the program activities to ensure robustness of the integration of cooperative elements into program activities. The project team will also include two leaders in the areas of undergraduate research experiences and cooperative learning: Heather Thiry and Karl Smith. They will serve as consultants on the project and will review the materials, evaluation plan, and the ARG Professional Development program.

• Heather Thiry, Research Associate in Ethnography and Evaluation Research (E&ER) at the University of Colorado, Boulder, has a Ph.D. in education and a background in psychology and women’s studies. Dr. Thiry has conducted extensive research on the benefits of undergraduate research for students in STEM fields and is an expert on diversity in science education. Along with her colleagues at E&ER, Dr. Thiry is currently writing a book documenting the personal, professional and intellectual benefits of undergraduate research for STEM students that is the culmination of a comparative, longitudinal study of undergraduate research at four institutions. She has published four refereed journal articles and made dozens of professional presentations. Dr. Thiry’s research interests also include the recruitment and retention of underrepresented groups into STEM fields, doctoral career paths, and professional socialization.

• Karl A. Smith, Cooperative Learning Professor of Engineering Education, Department of Engineering Education, Purdue University and Morse-Alumni Distinguished Professor of Civil Engineering at the University of Minnesota, is recognized as an expert in active learning strategies, cooperative learning, and faculty and teaching assistant development. Dr. Smith serves on the National Advisory Boards for the NSF-CLT Center for the Integration of Research, Teaching and Learning and for NAE’s Center for the Advancement of Scholarship on Engineering Education.

Each Hub leader commits to work with adopters at his or her Hub to implement the ARG model. Other responsibilities include: • Document effective practices and resources; publish on ARG website • Attend the annual summer ARG Professional Development program • Participate in regular meetings (Hub and teleconference) • Participate in the evaluation and assessment

The Hub leaders are as follows: • Maureen Biggers is the Assistant Dean for Diversity and Education at Indiana University and the

Program Manager for the NSF-funded A4RC. Her primary activities center on promoting success and retention of underrepresented students in computing; she has also served as a Co-PI on several other NSF projects including the STARS Alliance, GA Computes!, Collaborative Research: Increasing the Representation of Undergraduate Women and Minorities in Computer Science, and CPATH-EAE: Extending Contextualized Computing in Multiple Institutions Using Threads. She is on the leadership team for the National Center for Women in Technology.

• Nayda Santiago is an Associate Professor in the Electrical and Computer Engineering Department, University of Puerto Rico, Mayaguez Campus. Her main areas of interest are Performance

C14

Evaluation, Parallel Processing, and High Performance Computing Systems. She is currently collaborating with two NSF Engineering Research Centers: WIMS and CenSSIS. For the Wireless Integrated Microsystems (WIMS) Center, she is developing software for integrating subsystems developed for their cochlear implant test bed and microgas chromatograph test beds, taking into consideration low power requirements. For the Center for Subsurface Imaging (CenSSIS), she is working on porting hyperspectral imaging algorithms to hardware, specifically to FPGA and DSP platforms. She has been involved in undergraduate research since 2003.

• Manuel A. Pérez-Quiñones is Associate Dean in Residence and Director of the Office for Diversity Programs at the Graduate School, an Associate Professor in the Department of Computer Science, and a member of the Center for Human-Computer Interaction at Virginia Tech. His research interests include human-computer interaction, personal information management, user interface software, digital government, and educational uses of computers. He is a member of the Coalition to Diversify Computing, where he co-directs the national program Collaborative Research Experience for Undergraduates in Computer Science and Engineering (funded in part by a BPC grant). He serves on the editorial board for the Journal on Educational Resources in Computing. For 2008-2010 he has been included in the IEEE Computer Society Distinguished Visitor program. He has collaborated on several BPC projects, including the STARS Alliance.

• Dana Richter-Egger is Director of UNO’s new Math-Science Learning Center and Assistant Professor of Chemistry at UNO. He has a Ph.D. in chemistry and a background in mathematics and student leadership. He has an established a record of creative activity in science education and ambitious plans to improve science education at UNO for both STEM and non-STEM majors. He has directed undergraduate students in chemical research, integrated interdisciplinary research experiences into the general chemistry and intro geology courses at UNO, helped increase the number of STEM graduates at UNO by 43% (2003-2007) as co-PI on a NSF-STEP grant and serves as on the advisory committee of the Nebraska EPSCoR ‘Young Nebraska Scientist Initiative’.

6.2 Organization

Figure 2: Logic map for proposed work.

The logic map presented in Figure 2 describes the organization of the proposed effort. The columns on the right side of this figure illustrate the expected short- and long-term outcomes that motivate the project. The left side of this figure illustrates activities designed to achieve these outcomes. A closed-loop feedback model is used to achieve continuous quality improvement. The principal activities of this project, presented in the left column, are grouped as development, deployment, and evaluation and assessment. These are arranged as a closed loop continuous-quality-improvement model. The activities are designed to lead to the expected short- and long-term outcomes.

C15

Note that mechanisms are in place to collect data from seven institutions in the Computing Alliance for Hispanic-Serving Institutions (CAHSI). In addition, the ARG handbook has a chapter that presents an evaluation/assessment process and a framework for developing an evaluation plan to assess the effectiveness and impact of the undergraduate research experience. Although all institutions will be encouraged to collect data,. budget limitations prevent this grant from fully assessing long-term outcomes.

6.3 Year 1-3 Timelines and Deliverables • ARG Professional Development Level 1 Program

o Develop initial program: May 2009-Jul 2009 (Lead: Villa) o Review program: Jul 2009 (Lead: Smith with investigators) o Deliver workshops: Aug 2009, Aug 2010, Aug 2011, Aug 2012 (Lead: Gates with Smith) o Evaluate workshops: Aug 2009, Aug 2010, Aug 2011, Aug 2012 (Lead: Kephart) o Refine program: continual (Lead: Villa)

• ARG Professional Development Level 2 Program o Develop initial program: April 2010-Jul 2010 (Lead: Villa) o Review program: Jul 2010 (Lead: Smith with investigators) o Deliver workshops: Aug 2010, Aug 2011, Aug 2012 (Lead: Gates with Smith) o Evaluate workshops: Aug 2010, Aug 2011, Aug 2012 (Lead: Kephart) o Refine program: continual (Lead: Villa)

• Documentation and Dissemination o Develop initial materials for interactive ARG website to support faculty mentors and participants:

May 2009-Sep 2009 (Lead: Roach) o Launch website: September 2009 (Lead: Roach) o Contribute website materials: Sep 2009-May 2012 (Leads: Hub leaders) o Modify faculty resources for summer programs (Lead: Roach) o Publish papers in peer-reviewed journals and conference proceedings (Lead: all) o Meet with departments and institutions to identify funding venues to send faculty to summer

workshops: Sep 2009-May 2012 (Lead: Gates) • Videotaping of ARGs Hubs

o Videotape ARG meetings and activities and send to UTEP: Fall and Spring 2009-2012 (Leads: Hub leaders)

o Analyze and evaluate each of the Hubs: Fall and Spring 2009-2012 (Lead: Kephart) o Prepare evaluation report: April 2010, April 2011, April 2012 (Lead: Kephart with Thiry)

• Teleconferences o Schedule monthly teleconference with investigators and Hub leads and set agenda (Lead: Gates) o Respond to and document concerns of Hub leaders (Lead: Gates)