FLOSS in Software Engineering Education · 2018. 8. 6. · The SIGCSE 2016 Panel on “How to Use Open Source Software in Education” capitalized on the uses of FLOSS in education,

FLOSS in Software Engineering EducationAn Update of a Systematic Mapping Study

Moara Sousa BritoFederal University of Bahia

Salvador, [email protected]

Fernanda Gomes SilvaFederal University of Bahia


Christina von Flach G. ChavezFederal University of Bahia


Debora C. NascimentoFederal University of Sergipe

Aracaju, [email protected]

Roberto A. BittencourtState University of Feira de Santana

Feira de Santana, [email protected]

ABSTRACTContext: Free/Libre/Open Source Software (FLOSS) projects havebeen used in Software Engineering Education (SEE) to address theneed for more realistic settings that reduce the gap between soft-ware engineering (SE) courses and industry needs. A systematicmapping study (SMS) performed in 2013 structured the researcharea on the use of FLOSS projects in SEE. Objective: Update the2013 SMS with studies published in the last five years, classify-ing and summarizing them to discuss trends and identify researchgaps in the context of the use of FLOSS projects in SEE.Method:We retrieved and analyzed a set of 4132 papers published from2013 to 2017, from which 33 papers were selected and classified.We analyzed the new results and compared them with those fromthe previous SMS to confirm or discover trends. Results: The up-dated mapping summarizes the studies published in the last fiveyears, most of them in conferences. Our analysis confirmed trendspreviously observed for three facets (SE area, curriculum choiceand assessment type) and discovered new trends for other facets.Conclusion: Studies report the use of FLOSS projects in regular,comprehensive SE courses. The prevalence of experience reportsover solution proposals in the last five years may indicate thatresearchers are more concerned with the use and evaluation ofexisting proposals, although there are still opportunities for moreempirical work based on sound educational research methods.

CCS CONCEPTS• Social and professional topics→ Software engineering edu-cation; • Software and its engineering→Open sourcemodel;• General and reference→ Surveys and overviews;

KEYWORDSFree software, open source software, FLOSS development model,systematic literature reviews.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).SBES’2018, 2018, São Carlos, São Paulo, Brazil© 2018 Copyright held by the owner/author(s).ACM ISBN 123-4567-24-567/08/06.https://doi.org/10.475/123_4

ACM Reference Format:Moara Sousa Brito, Fernanda Gomes Silva, Christina von Flach G. Chavez,Debora C. Nascimento, and Roberto A. Bittencourt. 2018. FLOSS in Soft-ware Engineering Education: An Update of a Systematic Mapping Study.In Proceedings of SBES Conference (SBES’2018). ACM, New York, NY, USA,Article 4, 10 pages. https://doi.org/10.475/123_4

1 INTRODUCTIONFree/Libre/Open Source Software (FLOSS) projects have been usedin Software Engineering Education (SEE) to address the need formore realistic settings that could reduce the gap between soft-ware engineering courses and industry needs. The study of FLOSSprojects exposes students to large projects, best practices and pro-fessional tools for version control, build automation, and projectmanagement, while allowing them to experience the collaborative,social dynamics of team working on these large projects [23].

The most recent systematic mapping study (SMS) [4, 33] thatinvestigated the use of FLOSS projects in SEE, structured the re-search area according to nine facets and provided an overview ofinitiatives and research gaps based on work published prior to 2013[31]. This SMS identified active research communities, long-termprojects and venues for publication, and provided an overview ofobjectives, contributions and methods applied in 72 studies over14 years (1998-2012). This work provided a baseline for the schol-arship on SEE with OSP, describing previous work in terms ofresearch/scholarship type, software engineering area, curriculumchoice, OSP choice, control level, goals and assessment. Results fromselected papers showed a gap in the use of sound research methods,a lack of more active learning approaches as well a lack of educa-tional theory foundations. Most selected papers were publishedin conferences, describing either solution proposals or experiencereports on comprehensive software engineering courses based onthe traditional project method approach. Interestingly, this papershowed both the community interest of using OSP in SEE and theneed to improve scholarship to turn this area into a mature field ofresearch.

The SIGCSE 2016 Panel on “How to Use Open Source Software inEducation” capitalized on the uses of FLOSS in education, pointingout to barriers to its use in educational settings and the need for adebate to better take advantage of the collaborative ways to solvehard problems that open source communities and its ecosystemsbring [2].

https://doi.org/10.475/123_4

https://doi.org/10.475/123_4

SBES’2018, 2018, São Carlos, São Paulo, Brazil M. Brito et al.

(*) (*)

(*)

Figure 1: The update process, adapted from [31, 33].

Therefore, an updated SMS represents an important asset in anew research field that brings together FLOSS and SEE, where thenumber of studies and the maturity of empirical methods used areincreasing.

In this work, we present an update of the SMS on the use ofFLOSS projects in SEE [31]. We retrieved and analyzed paperspublished from 2013 to 2017, and classified them according to thefacets used in the original SMS. We analyzed the new results andcompared themwith those from the previous SMS to confirm trendsand discover new ones.

Our main findings point out both a confirmation of previousresearch trends as well as some changes that slowly move the fieldinto a more mature research topic. Confirmed trends are: 1) useof FLOSS to learn comprehensively on software engineering; 2)curriculum choice particularly focused on regular courses; and 3)assessment mainly based on reports and software artifacts. On theother hand, new trends point to: 4) an increased use of experiencereports and a reduction in solution proposals; 5) a growing interestof using FLOSS in software maintenance and evolution; 6) increaseduse of approaches where instructors have no control, and studentsinteract with the FLOSS community in real tasks; 7) less use ofpredefined FLOSS projects and more free project choice by students;8) a balance between goals of learning FLOSS and learning SEconcepts; and 9) the use of automated test suites as a relevantassessment type.

The rest of the paper is organized as follows. Section 2 presentsinformation about process steps performed and outcomes generatedin the context of the SMS update. Section 3 presents the results ofthe updated mapping and compares themwith results from the 2013SMS. Section 4 presents contributions and discloses opportunitiesfor future research.

2 THE SMS UPDATE PROCESSFigure 1 presents the process that guided our update to the SMSconducted by Nascimento et al. [31]. Steps marked with (*) werereused from the previous SMS. Their related outcomes (e.g., reviewscope and classification scheme), were the same as defined in [31].We used the StArt (State of the Art through Systematic Review)tool1 and spreadsheets to manage the data generated during theupdate process.

1http://lapes.dc.ufscar.br/tools/start_tool

The three research questions defined by Nascimento et al. [31]were also reused in this update: both the main question and the twosecondary questions. In this paper, we focus on the main researchquestion:

RQ1. How are Open Source Projects used in Software Engi-neering Education?

The review scope included the same search strings and inclu-sion/exclusion criteria defined by Nascimento et al. for paper search-ing and screening. Detailed information on the SMS protocol canbe found at the original SMS website2.

2.1 SearchWe conducted the search from March 22 to April 28, 2018. Thesearch strategy included the same scientific electronic databasesused by Nascimento et al. [31]: Engineering Village3, IEEE Xplore4,ACM5, Scopus6, Springer7 and Science Direct8. We retrieved 18,056papers, from which 13,924 were duplicates (various studies wereindexed by more than one digital library), leading to 4,132 papersselected for screening.

2.2 ScreeningFour inclusion criteria (IC) and 20 exclusion criteria (EC) definedby Nascimento et al. [31] were reused. For instance: (IC.1) Stud-ies that address the use of FLOSS projects to learn/teach SoftwareEngineering should be included, regardless of their application ineither SE programs or in other programs; (EC.1) Documents writtenin languages other than English should be excluded; (EC.2) Doc-uments whose full text is not available should be excluded; and(EC.3) Studies whose main content is not related to learning orteaching Software Engineering should be excluded.

We applied two filters during screening. First, the two first au-thors individually examined title and abstract of each primary study,and marked each study as either included or excluded. A third re-viewer compared the results, analyzed conflicts and took the finaldecision. In this first step, 39 studies were included.

2https://sites.google.com/site/dmcnascimento/mapping3http://www.engineeringvillage.com4http://ieeexplore.ieee.org5http://portal.acm.org6http://www.info.sciverse.com/scopus7http://www.springer.com8http://www.elsevier.com

FLOSS in Software Engineering Education SBES’2018, 2018, São Carlos, São Paulo, Brazil

Then, the two first authors reviewers reexamined the remainingstudies, skimming through introduction and conclusion. A thirdreviewer compared their results, and again, took the final decision,resulting in 33 primary studies selected for classification.

2.3 ClassificationThe SMS update reused the classification scheme proposed by Nasci-mento et al. [31]. Table 1 presents the nine facets defined for clas-sification purposes. Additional information about the facets areprovided in Appendix A.

2.4 Data extractionEach primary study accepted in the screening process was fullyread to collect the required data. We extracted information on title,authors, authors’ affiliation details, venue and year of publicationfor each selected paper, and additional information required toclassify it according to each facet. During data extraction, we alsolooked for information to characterize long-term projects and thecontinuity of research work.

3 THE UPDATED MAPPINGThis section presents the results of the updated mapping and com-pares them with results from the 2013 SMS.

3.1 Overview of study objectives andcontributions

We have identified several approaches in the selected papers. Oneof the main challenges of using FLOSS in SE education is selectingprojects for use in a course. Some studies address the choice ofsuitable projects, providing a predefined list of FLOSS [20, 32, 36].Gokhale et al. made available a list of seventeen projects compatiblewith students’ background. The selected projects ranged in sizefrom 5,500 to 10,500 lines of code. Smith et al. have listed ten opensource projects for exploring, experimenting and choice by students.In the study of Papadopoulos et al., students choose the projectfrom a list presented in the classroom. In some studies, studentsfreely choose the project they wanted to work during the course[5, 38]. In the study presented by Buffardi, teams were able to freelychoose a project to carry out their practices and apply the softwaredevelopment techniques learned in lectures. Van Deursen et al. letstudents choose a software system to study its architecture and todocument the system. In the study of Rekha and Adinarayanan,authors specify a set of criteria to aid the selection process. Groupsof up to six students undertook a search for projects hosted inthe SourceForge, GitHub and GoogleCode repositories. Diniz et al.previously established that students should work with the JabRefproject, an open source bibliographic reference manager.

Some studies present techniques for using FLOSS in SEE [10–12, 19, 32]. Dorodchi and Dehbozorgi used the Wilcoxon Mann-Whitney test, a non-parametric statistical hypothesis test, to com-pare two related samples. The result of their tests led to reject thenull hypothesis that confirmed the efficacy of the method. In theirstudy, Ding et al. subjected students to the technique of visualizingalgorithms, showing parameters, relevant variables, and visual rep-resentation of manipulated objects, as well as a formal descriptionof the algorithm. Students reported that software visualization tools

are easy to use and help understand software structure, behavior,and complexity. Diniz et al. used gamification to guide and motivatestudents to contribute to FLOSS projects. Gokhale et al. provided atutorial to help instructors identify appropriate software projects,design the set of tasks for students, and select useful tools. Pa-padopoulos et al. used a hybrid approach combining distance-basedproject learning, providing printed materials (e.g., books, tutorialsand articles), online resources and classroom lectures.

We identified tools that served to support the use of open sourceprojects in SEE [5, 32, 36, 38, 39]. Buffardi and Van Deursen et al.used the Github version control system to maintain project codeand track individual student contributions. In the study of Smithet al., student pairs were able to develop, test, and debug smallJava programs using Eclipse and to create UML class diagrams.The study of Villarrubia and Kim reports the experience of an opensource software community using the GitLab teaching environmentto support students’ collaborative software development activities.Papadopoulos et al. gave their students access to the FLOSSComand OpenSE virtual learning environments.

Other studies have described their assessment processes [6, 32,35, 39]. For Buffardi, project grade accounts for 70% of students’scores. They considered product usefulness, maintenance capacity,extension of software design, verification effectiveness, teamworkand final presentation. In [35], students were assessed from codepresentation, participation in communication channels, documenta-tion submitted at each stage, comments from peer review and con-tribution to the FLOSS community repository. To assess students,Villarrubia and Kim considered push, pull, and commit records thatstudents performed using GitLab. Finally, in [32] the assessmentconsidered the quality of produced work, documentation clarity andusefulness, number of bugs reported, complexity and effectivenessof developed code, and analysis depth and usefulness.

3.2 FLOSS projects in SEEThe main purpose of our SMS update was to discover how FLOSSprojects have been used in SEE (RQ1) over the last five years. Fig-ure 2 presents a map that combines Facet 1 - Software EngineeringArea, Facet 2 - Research Type and year of publication.

Facet 1 - Software Engineering Area. Table 2 shows how primarystudies addressed software engineering areas. Twenty-one studies(63.6%) deal with the general software engineering area. There wasa small decrease of 5.8%, when compared to the results from theprevious SMS [31]. Twelve studies (36.3%) were focused on particu-lar software engineering areas. Evolution/Maintenance leads thecount with six papers, followed by Development/Construction withtwo papers and Design/Architecture, Modeling/Analysis, Processand Testing, with only one study each. No papers were found thatrelate FLOSS projects with the knowledge areas of Requirements,Quality, Management and Configuration Management.

Facet 2 - Research Type. Table 3 presents the research methodsused in primary studies. Twenty-three papers fall into the experi-ence report category, representing 69% of the selected papers. Therewas an increase in this type of study, given that, in the previousmapping [31], this category represented only 19.4% of the studiesanalyzed. The category named solution proposal, the most popular


Table 1: Defined Facets [31]

Facet Description1 Software Engineering Area The SE topic(s) addressed in the study (software engineering in general, requirements, models and

methods, design and architecture, quality, testing, evolution and maintenance, development andconstruction, process, management and configuration management), based on the SWEBOK [2013].

2 Research Type The research approach used in the paper (experience report, case study, action research,experiment/quasi-experiment, survey, opinion paper, solution proposal, philosophical paper –adapted from Petersen et al. [33]).

3 Learning Approach The pedagogical approach that was applied together with FLOSS projects in SE courses – notpresented in this paper.

4 Assessment Perspective The perspective from which student learning is evaluated – not presented in this paper.5 Assessment Type The instrument used to assess student learning – see Table 11.6 Approach Goal Which goal in introducing FLOSS projects in SE courses was prevalent (either learning SE princi-

ples/concepts or learning to develop FLOSS).7 Curriculum Choice How the content was embodied in curriculum – see Table 12.8 Control Level How much control faculty/staff had on the FLOSS project – see Table 13.9 Project Choice How the FLOSS projects were chosen – see Table 14.

Figure 2: Software engineering area vs. year vs. research type.

Table 2: Studies classified by Facet 1 - Software Engineering Area

Area Studies #Software Engineering [3, 5–7, 9, 11–17, 21, 22,

26–28, 30, 32, 34, 35]21

Evolution/Maintenance [8, 19, 20, 24, 25, 37] 6Development/Construction [18, 39] 2Design/Architecture [38] 1Modeling/Analysis [10] 1Process [36] 1Testing [29] 1

Table 3: Studies classified by Facet 2 – Research Type

Category Studies #Experience report [3, 5–9, 12, 14, 16, 19, 20, 22, 24–

30, 36–39]23

Case study [10, 11, 13, 17, 18, 35] 6Experiment [32] 1Survey/questionnaire [15, 34] 2Proposal of solution [21] 1


Table 4: Studies classified by Facet 7 - Curriculum Choice

Category Studies #Extra activity [16, 30] 2Course [3, 5–9, 17, 20, 22, 25, 27, 28, 32, 35–38] 17Not specified [10–15, 18, 19, 21, 24, 26, 29, 34, 39] 14

in [31], was found only once in our update. This work reports thatempowering students to become socially responsible professionalsis a desirable outcome of SEE, and humanitarian free and opensource software (HFOSS) projects provide an opportunity for edu-cators to inspire students to face global humanitarian challenges(realistic settings) while learning about software engineering [21].While no Experiment/Quasi-Experiment had been found in theprevious SMS, this update found one work [32] in this category.Papadopoulos et al. reported the results after four years performingan instructional method that uses FLOSS projects as tools to teacha software engineering course.

3.3 Fitting FLOSS projects into students’activities

Figure 3 presents a map that combines Facet 8 - Control Level,Facet 7 - Curriculum Choice and Facet 9 - Project Choice to showhow FLOSS projects have been integrated into students’ activitiesover the last five years.

Table 4 shows the results for Facet 7 - Curriculum Choice. Moststudies still incorporate FLOSS projects into regular course curricu-lum (17 articles). In a different approach, two studies used FLOSS inextra activities, while 14 articles did not specify how they used it.

Two papers presented experiences in elective courses [8, 35].Rekha and Adinarayanan present the experience of preparing stu-dents to take on the roles and responsibilities of an incoming soft-ware engineer. Chen et al. report a preliminary study on the combi-nation of research and education in software testing.

Four articles describe experiences with undergraduate students[6, 12, 29, 39]. Fernandes and Barbosa explore the participation ofmaster’s students in FLOSS projects, mixing the learning contextsof formal and open/informal education. Ding et al. and Hislop andEllis report experiences in teaching and learning programming.Finally, two primary studies present approaches based on extraactivities [16, 30]. Morgan and Jensen describe two courses taughtat two universities, built around a model of communities of practice,and present lessons learned. Ellis et al. report an approach wherestudents are involved in an HFOSS project shared with the GNOMEAccessibility team and four academic institutions.

Table 5 presents the results for Facet 8 - Control Level. We clas-sified the studies according to three different categories on howmuch the faculty / staff controlled student activities. The majorityof the studies, corresponding to 48.4% (16 papers), used approacheswhere instructors had no control over the activities carried out bystudents. This scenario differs from the previous SMS, where therewas a balance between approaches where instructors had eitherinside control or no control over student activities.

The no control approach emphasizes the real-world experience,as students will need to interact with the project community, usersand developers. As an example of this approach, Fernandes and

Table 5: Studies classified by Facet 8 - Control Level

Category Studies #No control [6, 7, 9, 11, 13–18, 22, 25, 29, 32, 34, 36] 16Inside control [12, 21, 27, 35, 37–39] 7Full control [3, 5, 8, 10, 19, 20, 24, 26, 28, 30] 10

Barbosa used mailing list, chats, and forums to exchange ideas anddoubts or to share achievements by using the channels used by theFLOSS community. Moreover, the final part of the project requiresa work submission to relevant communities of each project. Inanother example, Fernandes et al. aimed to identify what should belearned about software development through active participationin FLOSS communities. In both examples, instructors only monitorstudents activities.

This SMS update identified only seven papers (21.2%) that fitinto inside control category, where there is no required interactionwith the community, and usually the instructor uses a project as anindependent branch. Dorodchi and Dehbozorgi describe the processof cloning an open source module and how students use structuredmodeling, such as data flow diagrams, functional decomposition,and UML diagrams, to reverse engineer the cloned module. Finally,we found ten studies characterized as full control, where the coreof the FLOSS community are the faculty themselves. Compared tothe original mapping, we had an increase from 12.5% to 30.3% ofstudies that adopted this category. This might indicate the growinginterest of academia for developing their own projects. It might alsomean that is is easier to choose a project known by their academicstaff. Ding et al. provide an example of this approach: they pickedout an open source project, some student-developed programs,and two software visualization tools, and then applied them in aprogramming course for students to evaluate the use of softwarevisualization in teaching and learning programming.

Table 6 presents the results for Facet 9 - Project choice, andit gives us insights into how selected studies deal with projectselection appropriate for FLOSS adoption in computing courses.We identified that students worked in free-choice projects on mostpapers where authors specified the way projects were selected,diverging from the previousmappingwhich claimed that, in general,students worked in predefined projects.

Eight papers whose project selection was free choice, such asVan Deursen et al. and Krutz et al., allowed students to chooseany real-world open source project for use in the SE course. Wenoted that Smith et al. and Pinto et al. gave students the freedom tochoose the projects. However, this choice process was supervisedby the instructor so that the chosen projects should meet certainconditions/constraints, e.g., project size, active design and num-ber of developers. The study presented by Smith et al. looked atstudents’ motives for choosing open source projects for use in amaintenance-focused software engineering course.

In seven studies, students worked on predefined projects. Forinstance, Morgan and Jensen chose the Ubuntu project for all theirstudents to work with, since such a project has several options forcontributors, including documentation, design, development, bugsand tests. Diniz et al. chose JabRef, a consolidated FLOSS project,


Figure 3: Curriculum choice vs. control level vs. project choice.

Table 6: Studies classified by Facet 9 - Project Choice

Category Studies #Predefined [8, 11, 15, 16, 27, 28, 30] 7Choice list [5, 17, 20, 22, 32] 5Free choice [9, 18, 25, 34–38] 8Not specified [6, 10, 12–14, 19, 24, 26, 29, 39] 10Does not apply [3, 7, 21] 3

where one of the authors is part of the project maintenance team,which facilitated instrumenting it.

Finally, we found five studies where instructors provided a listof projects for students. Among those studies, we may cite Gokhaleet al., that manually searched and evaluated about 1000 projectsfrom various FLOSS projects repositories and chose 17 projects thatwere compatible with their students’ previous experiences. As inthe previous mapping, we also identified studies that provided alist of FLOSS projects options, but students were free to choose aproject that was not in the list [18].

Unlike the previous SMS, Figure 3 shows that studies with fullcontrol level are not always predefined for students. We identifiedstudies with full control where project choice was either from a list[20] or chose by free choice [5].We identified 69.7% of the studies (23articles) with either no control or inside control. As in the previousmapping, most studies without a level of control, students had freechoice over their projects. Regarding the combination of Facet 7 -Curriculum Choice and Facet 8 - Control Level, we realized thatmost studies in regular courses have no control. In the previousSMS scenario, most studies with internal control were somewhatmore common in regular courses.

3.4 Assessing studentsTable 7 presents the results for Facet 6 - Approach goal. In this facet,concepts were developed and discussed in a systematic way. We

Table 7: Studies classified by Facet 6 - Approach Goal

Category Studies #Learning SE [3, 7–10, 12, 15, 17–20, 25, 28, 29, 32, 36, 38] 17Learning OSS [5, 6, 11, 13, 14, 16, 21, 22, 24, 26, 27, 30, 34,

35, 37, 39]16

Table 8: Studies classified by Facet 5 - Assessment Type

Category Studies #Exams [18] 1Reports [7, 9, 16, 17, 25, 32, 38] 7Software artifacts [5, 6, 8, 28, 30, 35, 39] 7Passing Test [10–12, 20, 24] 5Portfolio [3] 1None [13–15, 19, 21, 22, 26, 27, 29, 34, 36, 37] 12

observed that, in the last 5 years, 16 studies were classified in the“Learning about FLOSS projects in a Software Engineering Course”category, and 17 studies in the “Learning about SE concepts usingFLOSS projects” category. Thus, there was a balance between theapproach goals, differentiating our mapping from the previous one.Consequently, we perceived an interest of the academic communityin how students have learned about FLOSS projects. This interesthas been confirmed by Buffardi, who asserted that introducing SEstudents to FLOSS projects exposes them to challenges such asthose that they will face as professional software developers.

Table 8 describes the studies of Facet 5 - Assessment Type. Soft-ware artifacts (7 studies) and Reports (7 studies) are still the mainassessment instruments used by instructors in studies where learn-ing was explicitly assessed. However, from five primary studies, weidentified a new category, which we named “Passing Tests”, thatwas not mentioned in the previous SMS.


Figure 4: Approach goal vs. assessment type.

The map in Figure 4 shows an overview of how assessmentoccurs in the selected studies. Thus, we constructed this map bycombining the Facet 6 - Approach Goal with the Facet 5 - Assess-ment Type. The studies are described in Tables 7 and 8. We tried toidentify the student assessment perspective from the general goalsof the studies (Table 8). Most studies with the general goal of teach-ing SE concepts and principles used reports (6 articles), passing tests(3 articles) and software artifacts (2 articles). The study describedby Bowring and Burke used exams, and the study of Bowring andBurke used a portfolio. In studies with the goal of teaching OSS,predominantly no assessment was explicitly performed. From thesestudies, five papers used software artifacts and two other papersused passing tests. Only in the study of Ellis et al., they used reportsas assessment, although it was the predominant assessment typein the studies aiming to teach SE concepts and principles.

3.5 Distribution of publicationsIn the last five years, the interest of the research community on theuse of FLOSS projects in SEE has evolved. We present a temporalview of publications, main publication venues and the institutionsthat devote more effort to this subject.

Figure 5: Publications vs. year.

3.5.1 Temporal view. Figure 5 shows the distribution of publica-tions over the years. The green bars refer to the number of studies

Table 9: Main Publication Venues

Venue #FIE - Frontiers in Education Conference 3ICCSE - Intern. Conference on Computer Science & Education 3SIGCSE - ACM Technical Symposium on Computer ScienceEducation

2

CSEE&T - Conference on Software Engineering Educationand Training

2

ASEE Annual Conference and Exposition 2Journal of Computing Sciences in Colleges 2

identified in this present mapping. There has been a growing in-terest in this subject over the years. We notice an almost constantinterest after a 2009 peak and a sharp fall in 2010. There is a balancein the number of studies that were published in the last five years,with a small increase in 2017 compared to 2016.

3.5.2 Publication venues. Table 9 presents the distribution ofarticles by venue, considering venues that published two or moreselected papers. The complete set of publication venues (each withonly article from the 33 selected articles) can be found in our map-ping study website9. Most studies are still published in importantconferences, such as the Frontiers in Education Conference (FIE)and the ACM Technical Symposium on Computer Science Educa-tion (SIGCSE). In the previous mapping, the Annual Conferenceon Innovation and Technology in Computer Science Education(ITiCSE) was identified as the second largest publication venue.In this mapping, however, only one paper was published in thisconference. The International Conference on Computer Science &Education (ICCSE), not identified in the previous mapping, showsup with three papers included in this update.

Figure 6 shows how this topic has been addressed in the last fiveyears. As in the previous systematic mapping, conferences remainthe typical place to publish results (19 papers). Journals are thesecond most popular venue (9 papers). There was only one paperpublished in 2017 in both workshops and as books or book chapters.Finally, symposiums were also popular for publishing results, bothin 2014 (3 paper) and in 2017 (1 paper).

3.5.3 Active research communities. To learn which institutionshave devoted effort to studying student participation in FLOSSprojects as an approach to learn SE, we looked at affiliation detailsin our selected studies. Table 10 summarizes the communities thathad at least two selected publications in this mapping study.

Communities such as Drexel University and Western New Eng-land University, which are related to the Humanitarian Free andOpen Source Software project (HFOSS), continue to produce thelargest number of studies. These communities reeived contributionsfrom other communities that were not identified in the previousmapping. They are: Nassau Community College, Muhlenberg Col-lege, The College of New Jersey, Western Oregon University, Mora-vian College. Among the communities cited in Nascimento et al. ascontributors to HFOSS, only Bowdoin College was identified in ourmapping. Drexel University and Western New England Universitykeep very relevant to the research community, not only because of9https://moara.github.io/mapping/


Figure 6: Temporal view and publication venues.

Table 10: Active Research Communities

Institutions #Drexel University, US 7Western New England University, US 7University of Connecticut, US 4Nassau Community College, US 3Western Oregon University, US 3Northern Arizona University, US 2California State University, Chico, US 2University of Minho, Portugal 2The College of New Jersey, US 2Muhlenberg College, US 2Moravian College, US 2

the number of published studies, but also because of their involve-ment with a number of other communities. We have not identifiedTrinity College community-related publication within the past fiveyears, although they have published 12 studies in the previous map-ping. Regarding the Aristotle University of Thessaloniki and NorthCarolina State University, who respectively published four andthree papers in the previous mapping, we found only one study ineach of them. The University of Connecticut published two papersin 2012 (which were previously identified), and four papers between2013 and 2014. After that, we did not find other publications fromthis community about the use of FLOSS in SEE.

Two new communities appeared in the current mapping: Uni-versity of Minho, which published two studies related to collab-orative environments in teaching software engineering (FLOSSapproach), and Northern Arizona University, which attracted atten-tion by having two Brazilian authors (an Associate Professor at theNorthern Arizona University (NAU) and member of the ComputerScience Graduate Program at the University of São Paulo, and aPost-Doctoral Scholar at the School of Informatics, Computing andCyber-Systems at NAU, and Assistant Professor at the Federal Uni-versity of Technology - Paraná). They conducted research relatedto supporting newcomers in open source software developmentcommunities, including student involvement in FLOSS projects.

3.6 Long-term projectsAs we read the articles included in the mapping, we tried to capturethe maturity of each solution proposal in each study, based on howlong or how many times it had been applied. The scenario is similarto the previous mapping, with studies strongly related to others,and with few cases, since most of the identified studies continue toapply the proposed solution only once. Figure 7 presents studieswhose solution proposals were applied twice or more, and eachlinked set of nodes represents one identified project.

The largest identified project was HFOSS, with seven studies.Ellis et al. describes a workshop held to prepare faculty membersto deliver courses in which students participate in HFOSS softwaredevelopment projects. The 2014 study [16] and one 2015 study [14]describe student learning within the environment of an HFOSSproject. The other two 2015 studies ([15, 22]) report students’ opin-ions about the learning and impact of their participation in anHFOSS project. After one year without particular publications onthe use of open source projects in SEE, in 2017we identified two pub-lications with general information on Humanitarian Open SourceSoftware in Computing Education [21, 26] and information to fac-ulty and students on how to participate in the Humanitarian Freeand Open Source software OpenFE and OpenPath.

The University of Connecticut has developed a long-term project.The first studies published in 2013 [19, 20] describe experiences andlessons learned from using FLOSS to teach software maintenanceand evolution. The following papers, published in 2014 [36, 37],refer to the search, by instructors and students, of projects suitablefor use in SEE with a focus on maintenance and evolution.

The study of Fernandes and Barbosa is based on a pilot project inteaching and learning software engineering that has been carriedout for two years. The first findings were reported by Fernandeset al. in 2013.

In 2015, Buffardi described the Localized Free and Open SourceSoftware organization - LFOSS (an experimental approach to com-bine advantages of collaborating with industry with maintainingexisting open source projects), and reports initial findings fromsoftware engineering students’ involvement. In this study, he alsodescribes the Chico Open Source Consortium (COSC) as a group inChico, California, to foster collaboration between California StateUniversity-Chico (CSU Chico) students and local software profes-sionals in open source projects. In the following year, a new reportof a more realistic software development experience was published,continuing to focus on LFOSS [6].

4 CONCLUSIONWe presented an update to a systematic mapping study performedby our research group in 2013 to investigate the use of FLOSSprojects in Software Engineering Education. The first mappingselected 72 primary studies, published from 1998 to 2012, and cate-gorized them according to nine facets (software engineering area,research type, learning approach, assessment perspective, assess-ment type, approach goal, curriculum choice, control level andproject choice). The updated mapping selected 33 primary studiespublished from 2013 to 2017, and categorized them according tothe same nine facets. Moreover, we compared the results to provide


Figure 7: Long-term projects.

a comprehensive overview on how FLOSS projects have been usedin the context of SEE in the last 20 years.

There has been a steady but yet small body of research thataddresses the pedagogical use of FLOSS projects in SEE; it includes105 primary studies, developed by a small number of groups andresearchers, and mostly published in conferences on computerscience education. From 105 studies in 20 years, only one wasclassified as Experiment/Quasi-experiment in the Research Typefacet. Despite the increase in the number of experience reports,evidence shows that the research area is not mature yet.

The updated SMS is a valuable asset both to researchers inter-ested in the identified trends and gaps, and to instructors interestedin trying out their own experiences in their classes.

REFERENCES[1] 2013. Software Engineering Body of Knowledge (SWEBOK). (2013). www.

swebok.org/[2] J Bishop, C Jensen, W Scacchi, and A Smith. 2016. How to Use Open Source

Software in Education. In 47th ACM Technical Symposium on Computing ScienceEducation (SIGCSE ’16). ACM, New York, NY, USA, 321–322.

[3] J Bowring and Q Burke. 2016. Shaping Software Engineering Curricula UsingOpen Source Communities. Journal of Interactive Learning Research 27, 1 (2016),5–26.

[4] D Budgen, M Turner, P Brereton, and B Kitchenham. 2008. UsingMapping Studiesin Software Engineering. In Proceedings of PPIG 2008, Vol. 2. Lancaster University,195–204.

[5] K Buffardi. 2015. Localized open source collaboration in software engineeringeducation. In Frontiers in Education Conference (FIE), IEEE. IEEE, 1–5.

[6] K Buffardi. 2016. Localized open source software projects: Exploring realism andmotivation. In Computer Science & Education (ICCSE), 11th Intern. Conf. on. IEEE,382–387.

[7] D Castelluccia and G Visaggio. 2013. Teaching evidence-based software engineer-ing: learning by a collaborative mapping study of open source software. ACMSIGSOFT Software Engineering Notes 38, 6 (2013), 1–4.

[8] Z Chen, A Memon, and B Luo. 2014. Combining research and education ofsoftware testing: a preliminary study. In Proc. of the 29th Annual ACM Symposiumon Applied Computing. ACM, 1179–1180.

[9] V Cicirello. 2017. Student developed computer science educational tools assoftware engineering course projects. Journal of Computing Sciences in Colleges32, 3 (2017), 55–61.

[10] Y Ding, Y Hang, G Wan, and S He. 2014. Application of software visualizationin programming teaching. In Computer Science & Education (ICCSE), 9th Intern.Conf. on. IEEE, 803–806.

[11] G Diniz, Marco AG Silva, M Gerosa, and I Steinmacher. 2017. Using gamificationto orient and motivate students to contribute to OSS projects. In Cooperative andHuman Aspects of Software Engineering (CHASE), IEEE/ACM 10th Intern. Workshopon. IEEE, 36–42.

[12] M Dorodchi and N Dehbozorgi. 2016. Utilizing open source software in teachingpractice-based software engineering courses. In Frontiers in Education Conference(FIE), IEEE. IEEE, 1–5.

[13] H Ellis, M Chua, G Hislop, M Purcell, and S Dziallas. 2013. Towards a model offaculty development for FOSS in education. In Software Engineering Educationand Training (CSEE&T), IEEE 26th Conf. on. IEEE, 269–273.

[14] H Ellis, G Hislop, S Jackson, and L Postner. 2015. Team Project Experiences inHumanitarian Free and Open Source Software (HFOSS). ACM Transactions onComputing Education (TOCE) 15, 4 (2015), 18.

[15] H Ellis, G Hislop, SM Pulimood, B Morgan, and B Coleman. 2015. Softwareengineering learning in HFOSS: A multi-institutional study. In 22nd Annual ASEEConference and Exhibition, Seattle.

[16] H Ellis, S Jackson, D Burdge, L Postner, G Hislop, and J Diggs. 2014. Learningwithin a professional environment: shared ownership of an HFOSS project. In15th Annual Conference on Information Technology Education. ACM, 95–100.

[17] S Fernandes and LS Barbosa. 2016. Collaborative Environments in SoftwareEngineering Teaching: A FLOSS Approach. In European Conf. on e-Learning.Academic Conf. International Limited, 201.

[18] S Fernandes, MH Martinho, A Cerone, and LS Barbosa. 2013. Integrating formaland informal learning through a FLOSS-based innovative approach. In Intern.Conf. on Collaboration and Technology. Springer, 208–214.

[19] S Gokhale, T Smith, and R McCartney. 2013. Teaching software engineering froma maintenance-centric view using open-source software. Journal of ComputingSciences in Colleges 28, 6 (2013), 189–191.

[20] S Gokhale, T Smith, and R McCartney. 2013. Teaching software maintenancewith open source software: Experiences and lessons. In Frontiers in EducationConference, IEEE. IEEE, 1664–1670.

www.swebok.org/

www.swebok.org/


[21] G Hislop and H Ellis. 2017. Humanitarian Open Source Software in ComputingEducation. Computer 50, 10 (2017), 98–101.

[22] G Hislop, H Ellis, S Pulimood, B Morgan, S Mello-Stark, B Coleman, and CMacdonell. 2015. A multi-institutional study of learning via student involvementin humanitarian free and open source software projects. In 11th Annual Intern.Conf. on International Computing Education Research. ACM, 199–206.

[23] C Horstmann. 2009. Challenges and Opportunities in an Open Source SoftwareDevelopment Course. In Free and Open Source Software (FOSS) Symposium. 1–3.http://www.hfoss.org/symposium09/.

[24] V Krishnamoorthy, B Appasamy, and C Scaffidi. 2013. Using intelligent tutorsto teach students how APIs are used for software engineering in practice. IEEETransactions on Education 56, 3 (2013), 355–363.

[25] D Krutz, S Malachowsky, and T Reichlmayr. 2014. Using a real world project in asoftware testing course. In 45th ACM Technical Symposium on Computer ScienceEducation. ACM, 49–54.

[26] C Kussmaul. 2016. Experience Report: Guiding Faculty & Students to Participatein Humanitarian FOSS Communities. In Technology for Education (T4E), IEEE 8thIntern. Conf. on. IEEE, 224–227.

[27] CL Kussmaul, H Ellis, G Hislop, L Postner, and D Burdge. 2017. Board# 77:Helping Faculty & Students to Participate in Humanitarian Free & Open SourceSoftware: The OpenFE & OpenPath Projects. In 2017 ASEE Annual Conference &Exposition.

[28] B K MacKellar, M Sabin, and A B Tucker. 2015. Bridging the academia-industrygap in software engineering: a client-oriented open source software projectscourse. In Open Source Technology: Concepts, Methodologies, Tools, and Applica-tions. IGI Global, 1927–1950.

[29] D Mishra, T Hacaloglu, and A Mishra. 2014. Teaching Software Verification andValidation Course: A Case Study. International Journal of Engineering Education30 (2014), 1476–1485.

[30] B Morgan and C Jensen. 2014. Lessons learned from teaching open sourcesoftware development. In IFIP Intern. Conference on Open Source Systems. Springer,133–142.

[31] D Nascimento, R Bittencourt, and Chavez C. 2015. Open source projects insoftware engineering education: a mapping study. Computer Science Education25, 1 (2015), 67–114. https://doi.org/10.1080/08993408.2015.1033159

[32] PM Papadopoulos, I Stamelos, and A Meiszner. 2013. Enhancing software engi-neering education through open source projects: Four years of students’ perspec-tives. Education and Information Technologies 18, 2 (2013), 381–397.

[33] K Petersen, R Feldt, SMujtaba, andMMattsson. 2008. SystematicMapping Studiesin Software Engineering. In Evaluation and Assessment in Software Engineering(EASE’08), 12th International Conf. on. 68–77.

[34] G Pinto, F Figueira Filho, I Steinmacher, and M Gerosa. 2017. Training softwareengineers using open-source software: the professors’ perspective. In SoftwareEngineering Education and Training (CSEE&T), 30th IEEE Conference on. 1–5.

[35] S Rekha and V Adinarayanan. 2014. An Open Source approach to enhanceindustry preparedness of students. In Advances in Computing, Communicationsand Informatics (ICACCI), Intern. Conf. on. IEEE, 194–200.

[36] T Smith, S Gokhale, and R McCartney. 2014. Understanding students’ preferencesof software engineering projects. In Conference on Innovation & Technology inComputer Science Education. ACM, 135–140.

[37] T Smith, R McCartney, S Gokhale, and L Kaczmarczyk. 2014. Selecting opensource software projects to teach software engineering. In 45th ACM TechnicalSymposium on Computer Science Education. ACM, 397–402.

[38] A Van Deursen, M Aniche, J Aué, R Slag, M De Jong, A Nederlof, and E Bouwers.2017. A collaborative approach to teaching software architecture. In ACM SIGCSETechnical Symposium on Computer Science Education. ACM, 591–596.

[39] A Villarrubia and H Kim. 2015. Building a community system to teach collab-orative software development. In Computer Science & Education (ICCSE), 10thInternational Conference on. IEEE, 829–833.

A FACETS

Table 11: Assessment Type - Facet 5

Category DescriptionExams Students are assessed by means of written exams.Reports Students should write a report for assessment.Software artifact Students assessed through developed artifacts.Interviews Interviews are conducted to assess learning.Seminars Students assessed by performance in seminars.Exercises Students assessed by means of exercises.Surveys A survey is conducted to assess learning.Reflective essay A reflective essay is written.Presentations Students assessed by presenting their work.Participation Student’s participation in class, in community, in-

teraction with group or team.None No assessment instrument mentioned in paper.Does not apply Not related to activity where assessment is needed.

Table 12: Curriculum Choice - Facet 7

Category DescriptionExtra activity Students worked with FLOSS in extra activities,

e.g., internships.Capstone Project Students worked with FLOSS in a capstone

project.Course Working with FLOSS was a student’s assignment

in a regular course.Not specified The paper does not mention where, in the curricu-

lum, the approach is used.Does not apply Related to an experience where curriculum issues

are not declared.

Table 13: Control Level - Facet 8

Category DescriptionNo control Faculty/staff only monitor student’s activities in-

side the project. Students work with communityrequests and the community approves students’contribution.

Inside initiative/External approval

A new feature is proposed and built inside college,but later, it is submitted to community approval.

Inside control Faculty/staff branch the FLOSS code, prepare as-signments, and evaluate themselves the students’contribution.

Full control Project core development has been sustained byfaculty/staff.

Not specified No control level could be identified in the paper.Does not apply Work is not related to an experience where control

level is needed.

Table 14: Project Choice - Facet 9

Category DescriptionPredefined Faculty/staff decide the project students work.Choice list Students can choose any project from the list pro-

vided by faculty/staff.Free choice Students should seek and decide which OSP (from

their interest) they will work with.Not specified Paper does not mention issues related to project

choice.Does not apply Work is not related to an experience where choos-

ing a project is needed.

https://doi.org/10.1080/08993408.2015.1033159