The Pennsylvania State University

The Graduate School

AUTOMATING FEEDBACK-GENERATION FOR PROGRAMMING

ASSIGNMENTS VIA CROWDSOURCING

A Thesis in

Computer Science and Engineering

by

Rohan Shah

Ó 2020 Rohan Shah

Submitted in Partial Fulfillment of the Requirements

for the Degree of

Master of Science

December 2020

ii

The thesis of Rohan Shah was reviewed and approved by the following:

John Hannan Associate Department Head and Associate Professor Thesis Advisor

Danfeng Zhang Assistant Professor

Chitaranjan Das Distinguished Professor of Computer Science and Engineering Department Head of Computer Science and Engineering

iii

ABSTRACT

The importance of this project is to provide formative feedback to novice programmers

for programming assignments when they’re facing a bug. A Boolean feedback of test pass or fail

doesn’t help them resolve the issue and hence, formative feedback is important. Also, the

availability of feedback is another important factor, since with a high student to instructor ratio in

entry level computer science courses, it’s really hard for an instructor or TA to help every student

within a limited timeframe. Additionally, currently available solutions on providing feedback or

auto feedback generation are difficult to implement. So, this study investigates the use of

continuous integration tool to automate testing and to tackle the problem of providing formative

feedback for programming assignments, while keeping novice programmers in mind and with the

goal of implementing an unsophisticated solution, so an instructor can adapt it.

iv

TABLE OF CONTENTS

LIST OF FIGURES ............................................................................................................ v

LIST OF TABLES .............................................................................................................. vi

ACKNOWLEDGEMENTS................................................................................................. vii

Chapter 1: Introduction ....................................................................................................... 1

1-1: Background .......................................................................................................... 1 1-2: Motivation ............................................................................................................ 2 1-3: Problem Statement ............................................................................................... 3

Chapter 2: Literature Review............................................................................................... 4

Chapter 3: Method .............................................................................................................. 7

3-1: Technology Stack ................................................................................................. 7 3-2: System Design...................................................................................................... 8 3-3: Jenkins Setup ....................................................................................................... 9

3-3-1: Installing Jenkins CI and plugins ................................................................ 9 3-3-2: Jenkins-git project ...................................................................................... 12 3-3-3: Poll-email project ...................................................................................... 17 3-3-4: Automate feedback generation pipeline ...................................................... 19

3.4: Java Setup ............................................................................................................ 21 3-4-1: Compute Feedback for Programming Assignment in Java .......................... 22 3-4-2: Update Feedback based on crowdsourcing ................................................. 23

Chapter 4: Results ............................................................................................................... 24

Chapter 5: Discussion ......................................................................................................... 31

Chapter 6 Conclusion ......................................................................................................... 34

Bibliography ....................................................................................................................... 35

v

LIST OF FIGURES

Figure 3-1: System design. .................................................................................................. 9

Figure 3-2: General section of a Jenkins freestyle project. ................................................... 13

Figure 3-3: Source Code Management subsection of a Jenkins freestyle project. ................. 14

Figure 3-4: Shell script to test programming assignments and provide feedback................... 15

Figure 3-5: Shell script for poll email project ....................................................................... 18

Figure 3-6: Webhook integration on GitHub. ....................................................................... 20

Figure 3-7: Groovy script for automated feedback generation pipeline. ................................ 21

Figure 3-8: Feedback.java file to compute feedback for students based on their exception .. 22

Figure 3-9: UpdateFeedback.java function that updates feedback based on crowdsourced feedback ...................................................................................................................... 23

Figure 4-1: Student's java programming assignment file to be graded. ................................ 25

Figure 4-2: Shell script as a part of build step on Jenkins-git project. ................................. 26

Figure 4-3: Email response generated by Jenkins-git project. ............................................... 28

Figure 4-4: Email response generated from Jenkins pipeline for git-diff. .............................. 29

Figure 4-5: Git diff resulted page created from Jenkins pipeline. .......................................... 29

Figure 5-1: JSON representation of feedback.txt file. ........................................................... 31

vi

LIST OF TABLES

Table 3-1: Email configuration details for extended email plugin. ........................................ 11

Table 3-2: Editable email plugin configurations. .................................................................. 16

Table 3-3: Poll mailbox trigger configuration. ..................................................................... 17

vii

ACKNOWLEDGEMENTS

I would like to thank and acknowledge Professor John Hannan for his advice and patient

encouragement that helped me in writing and completing this thesis.

I would like to extend sincerest thanks to my family members and friends who supported

me and stood by me.

1

Chapter 1

Introduction

There has been heavy interest in the research field of auto generating feedback for student

programmers, as it is evident from the number of tools and papers that were reviewed in the

systematic literature review of automated feedback generation for programming assignments [2].

The number of tools considered were 101, while a total of 146 published papers were reviewed

on the same topic. Also, the one of the oldest papers of automating grading process can be dated

to 1964 [4], which shows that this problem has been in the industry for a while now.

The goal of this study is to provide novice programmers with partial feedback, rather than

simply providing them with yes or no as feedback. We like to consider partial feedback as

feedback that is not a complete solution but rather a hint, that conveys more information as

compare to what consoler output. The error message generated by consoler are not enough and

are generalized at times, and the solution to tackle this error varies from case to case. Hence,

partial feedback is required. Also, partial feedback can be thought as formative feedback, which

is defined in following section.

1-1: Background

Formative feedback is defined as: “information communicated to the learner with the

intention to modify his or her thinking or behavior for the purpose of improving learning” [7].

With the goal of generating formative feedback, this study started off with investigating how

Continuous Integration (CI) can be leveraged and utilized to automate feedback generation

process.

2

CI, as the name suggest, is a widely used tool in industry that helps integrate code of

different members of a software team. In industry, it is used for software building and testing,

since it allows to run a series of test code in order to improve software quality, based on preset

conditions. For example, a series of tests will be running automatically once a developer pushes

and merges code on GitHub. Also, GitHub is the version control and code hosting tool for

software development [15].

The next section describes the motivation behind the design of the proposed model and

then, chapter 2 explains the design and implementation style of a few models that have been

proposed in the past. One major factor that will be discussed is the difficulty level of adaptability

by an instructor for each model. Later in chapter 3, we discuss how each component of the

proposed model was chosen, followed by the detailed setup of CI tool i.e. Jenkins CI along with

detailed explanation of setting up backend i.e. reading and updating feedback using JAVA (as

discussed later, any programming language can be used here, as far as it can be compiled/run

from shell script). Following this, in chapter 4 - results, a running example is considered and used

to describe the flow of the proposed model and compute the feedback based on student’s example

java program. In chapter-5, we discuss the areas of the model that can be improved upon and

propose future directions for research that the current study can lead to. Finally, this paper is

concluded with ideas and advantages of the proposed model in chapter-6.

1-2: Motivation

One of the factors that was kept in mind while designing this model was if a student is

stuck on a problem and they come up with a solution, then use their experience (i.e. feedback in

this case) to help other students overcome the same problem, and hence the idea of

crowdsourcing the feedback. The feedback sources can differ, generally the one from instructor is

3

an ideal option but peer feedback could play a major role as well. This is evident from

HelpMeOut [5], which is a social recommender system, that collects feedback from peers and

their study was able to show that the model had a success rate of 47%, i.e. 47% of the total

feedback generated by peers was helpful. But as mentioned before, different types of model have

been proposed and implemented in the past, some sophisticated, some not so much (a few of

these models are discussed in detail in chapter-2, to understand the level of complexity to

implement solutions). So, this brings us to the second important factor of this study, i.e. to

propose and implement a solution that is easy to adapt for an instructor or a teaching assistant

(TA). Lastly, the functionality of running a series of test once a developer pushes and merges the

code on GitHub, motivated us to investigate and understand CI’s ability to automate the feedback

generation process. Since one of the functionalities of CI is unit testing, we realized it might have

a potential to automate formative feedback generation for programming assignments.

1-3: Problem Statement

This study proposes automating formative feedback-generation model via crowdsourcing

by using Jenkins CI for programming assignments and the assignments can be coded in any

programming language that has ability to compile or run via shell script.

4

Chapter 2

Literature Review

In this section, we will discuss various approaches for automated programming error

feedback that have been proposed in the past, and are discussed in [8]. This paper has mentioned

that there are five different approaches to solve this problem and they are as follows:

1. Additional Syntax Error Messages Feedback: In this approach, the models will output

additional error message with standard console output error message. One of the models that has

undertaken this approach is Sun-Java errors [9] but this is model hasn’t been implemented and the

effectiveness yet to be tested.

2. Solution Template Mismatches Feedback: Models that have been proposed using this

approach, require a sample program which is considered as a solution template. The sample

program consists of logic or semantics that students need to achieve and hence, it is compared

with student’s submission. Also, the model requires that the problem questions must be designed

with targeted goals and sub goals, and each sub goal has a template. Following two models were

designed based on this approach:

a. PROUST [10]: This model consists of goals and sub goals, where each sub goal is mapped to a

single or multiple solution templates. Each sub goals has a preset structure, by setting its input,

output, and the processing logic, which are designed by the instructor. Based on this template, it

will generate errors if there is a mismatch found between the solution template and the student’s

program code. It will then generate output hints as in the following example [2] [10]:

“The maximum and the average are undefined if there is no valid input. But lines 34 and 33

output them anyway. You should always check whether your code will work when there is no

5

input! This is a common cause of bugs. You need a test to check that at least one valid data point

has been input before line 30 is executed. The average will bomb when there is no input.”

b. Java Intelligent Tutoring System [JITS]: The design of JITS [11] model consists of a list of

programming decision trees which is used to assess a student's code against the trees. A

programming decision tree is a tree-based structure that has preset conditions and its

consequences. This decision tree is defined based on specific examples and problems and hence

limits an instructor to create a new problem, since the system cannot support it. One of the

examples mentioned in [8] helps understand the handling of JITS system to generate feedback for

loops. If a student code consists of loop, maybe a “while” or “for” statement, the system will

evaluate the student’s loop by checking for each section in depth and based on the consequences

set, it will evaluate next section. For instance, first it will test for conditional statement, then will

evaluate the update section, and so on, based on the design of decision tree. Lastly, JITS supports

feedback generation for typing error as well, as shown below:

“Would you like to replace smu with sum?”

3. Test Data Error Feedback: This approach is undertaken by tools that run tests on student’s code

on web browsers. In this approach, the instructor is required to design test cases and the web

application provides automation to test student’s code and provide feedback readily, based on the

tests designed by instructors. This design approach gives the benefit of hiding the test cases but

the downside is the instructor must design tests.

4. Assisted agent feedback [12]: The implementation goal of this approach is to generate

individual files based on individual’s activity. So, the file is generated by tracking and logging the

student’s online learning activity. One of the tools that uses this approach is TestMyCode [13].

Just like tools discussed in 3 that are designed on web-browser, this kind of tools stand out

because they provide smaller incremental tasks that results in a one big task. So, if a student is

stuck on an issue, the instructor can help them out by reviewing a small chunk compare to going

6

through a big chunk of code. The limitation of these kind of tools is, it’s not scalable for large

number of students.

5. Collaborative peer feedback: Lastly, the final approach is online discussion forums or social

networks such as Stack overflow [14]. Stack overflow is an online discussion forum that allows a

user to post their issue and other users who are expert in the same field can help resolve it. The

downside of this approach is the feedback is not guaranteed to available readily.

7

Chapter 3

Method

This section describes in detail what tools were considered, how the proposed automated

feedback generation pipeline was designed, and how-to setup Jenkins.

3-1: Technology Stack

There were two decisions that were needed to be made for implementing the automated

pipeline and they were: 1. the continuous integration (CI) tool and 2. the programming language

in which test programming assignments were to be coded. Also, it was decided that student’s

code will be uploaded to GitHub and hence use GitHub as the repository tool, since GitHub is

widely used at Penn State and elsewhere in the academia.

While deciding for CI tool, we considered XCode server and tried leveraging its

continuous integration (CI) with GitHub, but due to lack of documentation, unavailability of

third-party app integration, and difficult setup, we switched to Travis CI. One factor that was kept

in mind while selecting CI tool was to allow automating feedback-generation on as many

programming languages as possible, hence the switch to Travis CI was necessary, since the

investigation found that XCode Server is good for IOS or MacOS apps only. Next, upon

investigating Travis CI to leverage its CI with GitHub, we found that it was difficult to setup,

even for a simple build, and there were limited plugins available as compare to Jenkins CI, so we

decided to choose Jenkins CI as our continuous integration tool for this study. Jenkins CI is an

open source automation server for building, deploying, and automating any projects. It is easy to

use with documentation available for each step required in order to setup on a machine and offers

8

a range of plugins that were used in this study, such as email extension which allows user to

trigger (initiate) email based on user’s preset conditions like successful or fail build, etc.

Similarly, Jenkins CI has another plugin for integrating with GitHub which is very well

documented and easy to use. Also, all of the plugins that were leveraged in Jenkins CI are

explained in detail in next section, along with its limitations.

Next, in order to decide on programming language, we initially started using Swift and

Objective-C since we were investigating Xcode Server and later when we switched to Jenkins CI,

we decided to code test programming assignments in java due to its familiarity.

3-2: System Design

The following figure shows the overall system design of the proposed model. First, the

student pushes changes (programming assignment) to GitHub and based on the GitHub webhook,

Automated Jenkins pipeline will be triggered. The Automated Jenkins pipeline then runs the first

step which is triggering build for Jenkins-git project. Jenkins-git project then compiles and runs

student’s java code and based on the error detected, it calls getFeedback(). Next, the getFeedback()

java program returns the feedback to the Jenkins-git project, which further emails the feedback

along with results to students computed in Jenkins-git project’s shell file. Now the automated

pipeline takes control and it prepares the html template for git difference between current and

previous version, and emails it to the student as well. Finally, if the student thinks the feedback

received is not good enough and have a better feedback available that can help other students in

future, if they’re stuck at the same problem, the student can simply reply to the feedback email with

error name and better feedback. Once the student replies back to the received feedback email, poll

email project takes over the control and extracts the error name and student’s feedback, and further

calls the update feedback java program. Lastly, the update feedback java program adds the student’s

9

preferred feedback and upon next time other student faces the same issue, an additional feedback

will be provided to them (i.e. the preferred feedback added via student’s email to instructor).

3-3: Jenkins Setup

This section will discuss in detail the setup of the automated feedback generation pipeline

and projects using Jenkins CI.

3-3-1: Installing Jenkins CI and plugins

The setup for two freestyle projects and one pipeline used in Jenkins for the project will

be discussed in this section. They’re namely: Jenkins-git project, Polling email project, and

automated feedback-generation pipeline. Following are the terms that will be used in this section

in relevance to Jenkins [1]:

Figure 3-1: System design.

10

1. Project: A user-configured description of work which Jenkins should perform, such as

building a piece of software, etc.

2. Build: Result of single execution of project.

3. Pipeline: It is a suite of plugins which supports implementing and integrating continuous

delivery pipelines into Jenkins.

4. Plugin: An extension to Jenkins functionality.

5. Trigger: Conditions set by user to initiate a build.

6. Workspace: A disposable directory where work is done by a project or a pipeline.

Rest of the terms can be referred from Jenkins website [1]. Jenkins CI has a very well

documented guide on how to create an account and install Jenkins on the machine and run it on

localhost. As a reference, we installed Jenkins CI via homebrew on macOS using the following

command:

$ brew install jenkins-lts

and started it using the following command:

$ brew services start jenkins-lts

Lastly, it should be noted that the service can be terminated using the following command:

$ brew services stop jenkins-lts

Next, browse to http://localhost:8080 and it explains on how to unlock Jenkins and setup account.

After successful installation and unlocking of Jenkins CI process, the plugins needs to be installed

by clicking onto Manage Jenkins / go to plugin manager / available, find and install the following

plugins:

1. Email extension plugin: This plugin helps to select conditions to trigger (or initiate) email,

allows to set content to be sent via email, and lastly, it lets user specify recipients for the email to

be sent. Next, in order to configure this plugin, click onto Manage Jenkins / Configure System

11

and this page should show a tab for Extended email notification, where following configuration

details must be entered if using Gmail as a mailing tool:

Additional fields like default content type, default recipient list, default content, default pre-send

script, default post-send script are optional, but can be entered and if the same fields are left

empty at the configuration of freestyle project or pipeline, default values from this setting will be

used. Lastly, it must be noted that in order to use Gmail with this plugin, less secure app access

must be set to on, which can be found under Gmail account, /manage all Gmail accounts /

security. Hence it could be a potential security threat for the system.

2. GitHub branch source: This plugin allows to get source code from GitHub repository and a

webhook can be integrated to trigger builds with different criteria’s such trigger a build when a

user commits code to GitHub. More details about the configuration is explained later in this

section, as a part of creating a freestyle project.

3. Last changes: This plugin outputs a git diff and allows to compare git versions with previous

commit or of the depending on the revision id. Similar to GitHub branch source, more details

about configuration is followed.

Table 3-1: Email configuration details for extended email plugin.

SMTP server smtp.gmail.com

User name <your_email>@gmail.com

Password <your_password_for_gmail_account>

SMTP port 465

Charset UTF-8

12

4. Poll mailbox trigger plugin: This plugin allows user to poll or check for an incoming email that

matches either subject or the recipient or other different parameters and based on the set criteria,

it triggers a build. Again, configuration details are explained later in this section.

Also, plugins related to pipeline would’ve already been installed as a part of installing

and unlocking Jenkins process, since during the process, Jenkins allows user to install a custom or

recommended range of plugins.

3-3-2: Jenkins-git project

Once plugins were installed and email extension was configured, a new freestyle project

was created from new item. This section explains the setup for first freestyle project called

Jenkins-git. Each freestyle project has six components that can be configured, as explained

below:

1. General: This subpart has a field for project description and a few other checkboxes like

discard old builds, this project is parametrized, etc. Since none of the functionalities offered in

general were of interest to us, so we will skip to next part and leave all checkboxes unchecked, as

shown below:

13

2. Source code management: As the name suggest, this section allows user to configure git

repository. Select Git in this subsection and enter repository URL and credentials (if any) along

with the git branch name in branch specifier field, as shown below:

Figure 3-2: General section of a Jenkins freestyle project.

14

3. A Build triggers: No configuration is required for this step since Jenkins 3rd component i.e. the

setup of a pipeline, described in at the end of this section, will discuss the setup of a pipeline that

will be responsible for triggering this build. This section allows user to trigger a build based on

predefined 5 criteria by default, and 6 criteria, if Poll mailbox trigger plugin is installed (also one

possible way to verify if Poll mailbox trigger plugin was successfully installed or not while

unlocking Jenkins). It should be noted that another freestyle project described (Jenkins second

component for this study) in next section will be using Poll mail trigger, which allows user to

trigger a build if email’s subject is matching or other preset criteria. So as a reference, this is the

subsection where different criteria’s for triggering a build is set and Jenkins offer various criteria

like triggering build from a different script, building periodically, etc.

Figure 3-3: Source Code Management subsection of a Jenkins freestyle project.

15

4. Build environment: This section allows operations that are relevant to build environment such

as deleting workspace before build starts or passing a secret credential file, etc. No checkboxes

are checked in this section.

5. Build step: In this subsection, we ran a shell script that was capable of calling programming

assignment java file that was pushed to GitHub by a student and extracted the output for

exception or error handling. Furthermore, the output was passed to another java file that provided

formative feedback based on the student’s programming assignment error. So, under add build

step, execute shell was selected and a shell script was included. Note, it is ideal to upload the

shell script in github and the following command could be used in the Jenkins shell to call the

“<scriptname>.sh” that exist on github:

bash -ex <scriptname>.sh

As a reference, the following is the shell script that is called once a build is triggered by github’s

POST request based on a student pushing their code:

The script is written to compile student’s programming assignment and feedback generation java

file coded by the instructor, and run tests and generate hints for student. The detailed logic of this

script is discussed in Results section with an example.

7. Post build actions: As the name suggest, this section allows user to run various actions after the

build has executed. In this section, we selected following two actions:

Figure 3-4: Shell script to test programming assignments and provide feedback.

16

a. Publish Last Changes: If last changes plugin was successfully installed, in add post-build

action, public last build changes was selected and “Since” was set to “Previous Revision”. (This

step is optional as we will see in chapter 3.2.1.3 on how to include the same logic as a part of

pipeline).

b. Editable Email Notification: This section allowed us to email results along with feedback to the

student once the shell script was successfully executed and results were computed. Following

configuration was used:

It’s important that Project Reply-To List has an email listed, since it’ll be used for second phase

for collecting hints from students in order to improve the feedback model. Also, the reason

default content has the start-here and end-here configuration is because that’s the output we

emailed to students and is of interest in the shell execution output. And as it can be seen in the

shell script, there is start-here and end-here enclosed in the html tags, which are supported by

Default-Content.

Table 3-2: Editable email plugin configurations.

Project Recipient List <recipient_email>

Project Reply-To List <instructor’s_email>

Content Type HTML (text/html)

Default Subject Subject of the email

Default Content ${BUILD_LOG_EXCERPT,

start="\\b(start-here)\\b", end="\\b(end-

here)\\b"}

Attached Build Log Do not attach build log

17

3-3-3: Poll-email project

In this section, the setup for polling (checking) unread emails sent by a student to

improve feedback map is explained. Note that the actual business logic for improving feedback

map is in a java file and explained in the Results section. So, a freestyle project was created from

new item and following are the sections that were configured:

1. General: Added description of this project and left all checkboxes unchecked

2. Source Code Management: Followed the same process as source code management section of

Jenkins-git project.

3. Build Triggers: Poll mailbox trigger was selected with following configuration:

The password in the above configuration is the instructor’s or TA’s email password or the

account which will be used for monitoring improved hints students will email to the instructor or

TA. In the advanced email properties, there’re other predefined variables which can be found as a

part of poll mailbox trigger plugin’s documentation but the ones used in here are specifying that

the email’s folder name should be Inbox and subject of the email must contain

Re:<Subject_of_email_sent>. Subject_of_email_sent implies the subject set in the configuration

of chapter Jenkins-git project and step 6. So, the idea is to ask a student to reply to the email they

Table 3-3: Poll mailbox trigger configuration.

Host imap.gmail.com

Username <instructor’s_email>

Password <password>

Advanced email properties folder=INBOX

subjectContains=Re: <Subject_of_email_sent>

Schedule H/5 * * * *

18

received as a feedback, if they think they’ve better hints that can help other students. Hence,

“Re: ” is added to the subject line. The idea of improvement of feedback map by receiving

students’ input is discussed more in detail in the results section. Lastly, the schedule field has H/5

**** which implies that every five minutes, the email will be polled to check for matching

subject or any other set criteria. The period can be changed and customize based on needs and

more details on how to set the period can be found by clicking on ‘?’ next to Schedule field on

Jenkins.

4. Build Environment: No changes required, all checkboxes are left unchecked.

5. Build Step: Execute shell was selected as a build step and the following shell script was added:

The reason for including git checkout master and git pull is to have a most updated

version of feedback.txt from GitHub to Jenkins workspace and then add extra fields in the .txt file

using UpdateFeedback.java file. Also, the goal was to compile UpdateFeedback.java file and run

the java file with variables $error and $fdb by extracting variables from the email sent by student

to instructor. More details on how the two fields were extracted is explained in results section

with an example.

6. Post-build actions: The following needs to be added from add post build actions:

configuration is required for this step.

7. Git publisher: This post build action allows to push changes into git from shell. So, after

selecting git publisher from add post-build actions, following changes were made:

o Check ‘Push only if build succeeds'

git checkout master git pull javac UpdateFeedback.java output=$(java UpdateFeedback $error $fdb) git add feedback.txt git commit -m "updated feedback text" git push

Figure 3-5: Shell script for poll email project

19

o Branch to push: master

o Target remote branch: origin

3-3-4: Automate feedback generation pipeline

In this section, we will discuss how to setup Jenkins pipeline which will be responsible

for triggering a build based on a GitHub change, building Jenkins-git project, and lastly email the

student with git diff (Git difference of two versions of commits) of current and previous revisions.

From new item, pipeline was selected and we named it as Automated feedback generation

pipeline. Before configuring the pipeline, GitHub needs to be integrated with Jenkins to receive a

POST request when a change is pushed on GitHub to trigger a build. Under Manage Jenkins, go

to Configure System and after scrolling to the bottom of the page, there’s a section for GitHub

Pull Request, where we entered the git repository link in Published Jenkins URL. Next, similar to

project, there are various configurations available for Jenkins pipeline as explained below:

1. General: In this section, we entered the description of the pipeline and rest was default.

2. Build Triggers: In order to receive a POST request from github to Jenkins, webhook needs to

be integrated in the git repository that is used in this project. A webhook is a notification (http

POST request in this case), that allows integration between two applications. So, under git

repository, go to /settings/Webhooks/ and select add Webhook. Once selected, enter the Payload

URL in the following format: <Jenkins_server_url>:<port #>/github-webhook/

and as shown below:

20

3. Advanced Project Options: No configuration change is required.

4. Pipeline: This section allows user to provide a groovy script which has steps to build various

projects. The following groovy script was included:

Figure 3-6: Webhook integration on GitHub.

pipeline { agent any stages { stage('One') { steps { build 'jenkins-git' git 'https://github.com/rohanshah18/jenkins' script { def publisher = LastChanges.getLastChangesPublisher "PREVIOUS_REVISION", "SIDE", "LINE", true, true, "", "", "", "", "" publisher.publishLastChanges() def htmlDiff = publisher.getHtmlDiff() writeFile file: 'build-diff.html', text: htmlDiff emailext ( subject: "Jenkins - changes of ${env.JOB_NAME} #${env.BUILD_NUMBER}", attachmentsPattern: '**/*build-diff.html', mimeType: 'text/html',

21

The script is designed such that it first triggers a build for Jenkins-git project that was

setup in section 3.2.1.1 and after that, based on the git repository mentioned, it will get git

difference of current version and previous version and write it to a new html file. Once the html

file is ready, it will initiate an email to the recipient mentioned in last line with a link to the html

file that was just created, so that the recipient (student) in our case can see git diff of current and

previous git versions.

3.4: Java Setup

It must be noted that any programming language that can run from shell script can adapt

this model of automating feedback generation via crowdsourcing, since Jenkins and the shell

scripting is the requirement. But in order to show the working model, the following two java files

are needed, which essentially helps to compute feedback for students and second one is to

improve hints based on student’s input. Also, feedback.txt file must be created which contains

error name and feedback in the following format:

<errorName>; <feedback>

java.lang.ArrayIndexOutOfBoundsException; Did you try to access an array's ith element whose value is not defined yet?

body: """<p>See attached diff of build <b>${env.JOB_NAME} #${env.BUILD_NUMBER}</b>.</p> <p>Check build changes on Jenkins <b><a href="${env.BUILD_URL}/last-changes">here</a></b>.</p>""", to: "rohanshah018@gmail.com") } //end script } } } }

Figure 3-7: Groovy script for automated feedback generation pipeline.

22

3-4-1: Compute Feedback for Programming Assignment in Java

Feedback.java file as shown below is called by Jenkins-git project if the student’s

submission has run time error exception. Currently the function is designed to read each line of

feedback.txt and if the student’s run time exception matches the error in feedback.txt file, it will

store the corresponding hints in a temporary array-list and return them. It must be noted that for

same error, different solutions are possible and hence multiple hints.

import java.io.*; import java.util.ArrayList; public class FeedBackMap { private static ArrayList<String> getFeedback(String error) { error = error.trim().toLowerCase(); ArrayList<String> hints = new ArrayList<String>(); String line; try { BufferedReader bufferReader = new BufferedReader(new FileReader("feedback.txt")); while ((line = bufferReader.readLine()) != null) { String[] words = line.split(";"); if(words[0].toLowerCase().equals(error)) { hints.add(words[1]); } } } catch (FileNotFoundException ex) { ex.printStackTrace(); } catch (IOException ex) { ex.printStackTrace(); } return hints; } public static void main(String[] args) { ArrayList<String> hints = getFeedback(args[0]); for (int i = 1; i <= hints.size(); i++) { System.out.println(i + ". " + hints.get(i - 1)); } } }

Figure 3-8: Feedback.java file to compute feedback for students based on their exception

23

3-4-2: Update Feedback based on crowdsourcing

UpdateFeedback.java is called by Poll email Jenkins project from the shell script in the

build step with two arguments, namely error and feedback, that student’s think would’ve worked

for them. Once this method is called, it writes to the feedback.txt files by adding the error and the

hint while maintaining the format of the .txt file with <errorName>; <updated_fdb>. Finally it

returns the success message that the feedback.txt file was updated or in the case of error, it throws

error handled by try-catch mechanism.

This component helps us to improve our model overtime and hence the feedback map is learning

based on crowdsourced feedback.

import java.io.FileWriter; import java.io.IOException; public class UpdateFeedback { public static void main(String[] args) { System.out.println(updateFeedback(args[0], args[1]+ "\n")); } public static String updateFeedback(String error, String hint) { try{ FileWriter fileWriter = new FileWriter("feedback.txt", true); String fileContent = error + "; " + hint; fileWriter.write(fileContent); fileWriter.close(); } catch (IOException e) { // exception handling } return "the error: " + error + " and the feedback: " + hint + " has been successfully added."; } }

Figure 3-9: UpdateFeedback.java function that updates feedback based on crowdsourced feedback

24

Chapter 4

Results

This chapter shows a running example and the results achieved, so an instructor can

estimate the amount of work involved and the simplicity of the proposed solution. Also, the java

setup from section 3.3 is a requirement along with following files and scripts mentioned in this

section.

The following is the Student.java file that has four different functions namely isEven,

reverseString, sum2, and failFunc. It should be noted that failFunc() is coded incorrectly and

throws a runtime exception of ArrayIndexOutOfBounds, to understand the working of the

proposed model.

Public class Student { // checks if an int is even public static boolean isEven(int num) { return (num % 2 == 0); } // Reverses a String public static char[] reverseString(String S) { char[] s = S.toCharArray(); int left = 0, right = s.length - 1; while (left < right) { char tmp = s[left]; s[left++] = s[right]; s[right--] = tmp; } return s; } // Sum two int private static int sum2(int a, int b) { return a + b; } // Function fails with error ArrayIndexOutOfBounds private static void failFunc(int[] a) { int i = a[1]; } public static void main(String[] args) throws Exception { try { int[] arr = {1}; System.out.println(isEven(Integer.valueOf(args[0]))); System.out.println(reverseString(args[1]));

25

Once the changes are pushed to GitHub and assuming above is the latest version of

Student.java, GitHub will send a POST request to Jenkins CI and trigger a build for pipeline

discussed in section 3.2.1.3. Based on the logic in the pipeline, it will first build Jenkins-git

project with the following shell script included in build step:

System.out.println(sum2(Integer.valueOf(args[2]), (Integer.valueOf(args[3])))); failFunc(arr); } catch (Exception e) { System.out.println(e.getClass().getName()); } } }

Figure 4-1: Student's java programming assignment file to be graded.

javac Student.java javac FeedBackMap.java testNum=15 rem=$(($testNum%2)) testString='Apple' SumNum1=2147483647 SumNum2=1 output=$(java Student $testNum $testString $SumNum1 $SumNum2) arr=() res=() while read -r line; do IFS='.' read -a array <<< "$line" if ([ ${array[0]} == "java" ]) then res4="failFunc is failing. Hints: " res4+=$(java FeedBackMap $line) else arr+=("$line") fi done <<< "$output" if ([ -n "${arr[0]}" ] && [ ${arr[0]} == "true" ] && [ $rem == 0 ]) || ([ -n "${arr[0]}" ] && [ ${arr[0]} == "false" ] && [ $rem == 1 ]) then res1="isEven test has passed for int input=$testNum." else res1="isEven test has failed for int input=$testNum." fi if ([ -n "${arr[1]}" ] && [ ${arr[1]} == "elppA" ]) then res2="reverseString test has passed for string input=$testString." else res2="reverseString test has failed for string input=$testString." fi

26

As it can be seen in the above shell script, it first compiles the Student.java file and

FeedBackMap.java file (from chapter 3). Next, it runs the Student.java file with input parameters

set in the shell script and passed along with the command:

$(java Student $testNum $testString $SumNum1 $SumNum2)

After the Student.java file runs, it can be seen that the main method when initiated, will

run all four functions that were supposed to be coded by students and it outputs the result one

after the other. So, in this model, we’re leveraging the console output and reading it in the shell

and storing it in a variable. Next, this output is compared line by line and following two things are

done:

1. Checks for runtime exception, since in the Student.java, we’re handling all exceptions using try

catch and, in the catch, the exception name is printed. In Jenkins shell, the runtime exception is

detected by the following pseudo code:

Psuedo code:

read each line of the output:

split the line by “.” and store it in temporary array called words:

if words[0] is equal to “java”, error is detected:

call FeedbackMap with error-name

if ([ -n "${arr[2]}" ] && [ ${arr[2]} -gt -1 ]) then res3="sum2 test has passed for inputs: $SumNum1 and $SumNum2." else res3="sum2 test has failed for inputs: $SumNum1 and $SumNum2. Hint: Did you check for maximum integer value Java int can handle?" fi echo "start-here\n <br>${res1}</br> <br>${res2}</br> <br>${res3}</br> <br>${res4}</br>\n end-here\n"

Figure 4-2: Shell script as a part of build step on Jenkins-git project.

27

After the FeedbackMap.java is called with error-name, as per the logic discussed in

chapter 2 for FeedBackMap.java, it will print hints to the console for the corresponding error.

Also, the logic for detecting run-time exception can be improved overtime, but for simplicity, we

assumed no function is returning an output with the following format: java.<some_string> i.e. the

first word right before “.” is “java”.

2. If no runtime exception is detected, the output is stored in an array, which will be used for

testing purpose.

Once the output is stored in an array, from the shell script in figure 4-2, it can be used to

design test cases in the shell itself. For example, in the shell script, there’re three test cases coded

by if-else statements, which are designed to test functionalities for isEven(), reverseString(), and

sum2() of Student.java file. After the output from calling FeedbackMap.java is stored in a

variable $res4 and the if-else statements in shell script has been executed and stored in variables

$res{i} (i ranging from [1:3]), the shell script echoes them to the console and is surrounded by

HTML tags with “start-here” and “end-here”. The reason it is printed to the console in HTML

format is because in chapter 3, where Jenkins-git was setup, step#5 had post-build action, which

configured email settings and as discussed there, the email will include all contents after <start-

here> tag and until <end-here> tag. Also, the content-type was set to HTML and hence the reason

why the echo statement in shell script is surrounded by HTML table tags i.e. <br> and </br>. So,

at the end of execution of this script, an email will be triggered and delivered to the recipient list,

configured in the editable email notification section. Also following is the email response

received by the recipient list in our working model:

28

Based on the email response, the student can infer that isEven() and reverseString()

functions were coded correctly, although sum2 function is easy to design, but one of the

requirements to return sum of two integers as an integer clearly failed, when one of the integers

passed as an argument was the maximum value of an integer that java can hold. Hence, the

feedback in test results for #3 is along the lines. The same test can be designed in java (say in a

new file testHandling.java) and the error can be added to the feedback.txt file with appropriate

error naming convention. The only change required to support such logical errors (compare to

runtime errors) in testHandling.java, is appropriate error detection in shell script. As discussed

earlier in this section, the shell script detected runtime errors by detecting if the first word before

“.” is “java”, but passing the logical error name in testHandling.java file and running the error

across the feedback.txt file for matching error name, hints can be computed for logical errors as

well. Hence, this approach is capable of handling run-time errors as well as logical errors in the

language desired (as far as it callable by shell script).

Now, if we recall, building and running Jenkins-git project was step 1 that was initiated

by automated feedback generation pipeline. So, based on the pipeline script from figure 3-6, it’ll

now create a new html file that contains git diff of previous and current version and email it to the

recipient list, as discussed in chapter 3.2.1.3 while setting up pipeline. Following is an example of

email received for git diff:

Figure 4-3: Email response generated by Jenkins-git project.

29

And this is an example of git diff page resulted from clicking the link in the above email and was

generated by the second step in pipeline:

Also, it should be noted that the average time taken from triggering the build for the

pipeline (by GitHub, after a code change is pushed) to receiving both, feedback on test results and

the git diff via email, is around 16s. This time will vary depending on the size of the files and

number of tests that’ll be ran for different cases, but it is important to note that compare to

receiving feedback from TA/instructor, receiving feedback from the proposed model is

significantly lower.

Figure 4-4: Email response generated from Jenkins pipeline for git-diff.

Figure 4-5: Git diff resulted page created from Jenkins pipeline.

30

Last phase of the proposed model and the most important one, is the ability of the model

to improve overtime via crowd-sourcing (part of the solution was inspired by Socure inc.’s [9]

engineering). When a student receives feedback in the form of email, as shown in figure 4-3, if

they think the feedback wasn’t enough and could come up with a better feedback that helped

them solve the problem, they can simply reply to the email with a better feedback (that can

potentially help other students in future) for the given error. Since the instructor/TA’s email is

continuously polled by the Jenkin’s poll email project that was discussed and setup in chapter 3,

the UpdateFeedback.java file will then be called and the updated feedback in will be registered in

the feedback.txt file. This makes the proposed model self-sustaining, and it has the ability to

improve over time with minimal TA/instructor’s intervention. Also, as mentioned before in

chapter 3, the format for the feedback received from student matters and hence they must reply

back to the email in the following format:

error=<error_name>

feedback=<updated_feedback>

i.e. with key and value pairs, where error and feedback are keys and they’re used as variables in

the shell script of poll email project. It should be noted that we faced bugs in poll email plugin

and hence, might require reconfiguration. After the updated feedback is registered, if another

student stumbles upon the same problem, they now have better library of hints available.

31

Chapter 5

Discussion

There are a few areas that this model can be improved upon and some future directions

that this study can lead to, so this section will focus on that. Currently, the model stores hints and

error names without any order or format; so instead, they could be stored in a JSON file with each

error and the hints as a JSON object, as shown below:

This change can improve the lookup time for getting hints, since currently, the brute force

solution is traversing through each line of error and hint, and checking if there’s a match. Clearly,

if the list of errors and the corresponding hints are large, the solution is not scalable. Additionally,

implementing JSON in java or Scala projects can be done in a simpler way by using maven

dependencies. Also, the getFeedback code, the updateFeedback code, and the unit/functional tests

designed for programming assignments can be hidden behind a client and students can simply use

the api or client to call and get the feedback for their programming assignments. This approach

{ "task1": { "java.lang.NullPointerException": { "hints": [ "Did you try to use a variable that was not initialized before?" ] }, "java.lang.ArrayIndexOutOfBoundsException": { "hints": [ "The array is indexed at 0 in java", "Did you try to access an array's ith element whose value is not defined yet?" ] } } }

Figure 5-1: JSON representation of feedback.txt file.

32

will allow easy implementation/setup of project for an instructor/TA as well as the ability to hide

unit or functional test from students.

Next, the feedback.txt file stores error and hints, but adding the frequency, i.e. the

number of times a particular hint was successful in helping a student to resolve their issue, will

make this model more accurate. That is, the hints received by students’ overtime will be more

accurate. The frequency count implementation can be achieved by adding an extra variable in the

feedback email provided to students asking if the hint was helpful, let’s say variable name

hintHelpful. Then, the email response from student will then look like the following:

error= java.lang.NullPointerException

hintHelpful=Yes

feedback=NA

The reason feedback variable is left empty or set to NA is because feedback in here

represents the better feedback that the student thinks would’ve helped him resolve the situation,

but since the feedback received via the email, generated from the pipeline was enough, the

hintHelpful variable is set to yes. On the backend, on receiving hintHelpful=Yes, the

updateFeedback function will simply update the frequency count of the corresponding hint, by

adding 1 to it. After the frequency count is updated to +1, next time the getFeedback function is

called on the same error, the hints will be returned in descending order of frequency. So, the

chances are, the first few feedbacks will be enough to solve their bug. Hence, adding a frequency

count in the model will allow the model to learn over time.

Next, as discussed in chapter 3-2-1 while configuring email settings for extended email

plugin in Jenkins, there is a potential security threat for Gmail users, since in order to use the

extended email plugin successfully, “less secured apps access” (found under manage gmail

accounts/security) was set to ON, and this is not recommended by Google. So, finding an

33

alternative plugin or resolving this issue is of utmost importance, especially with growing

cybersecurity threats.

Also, in chapter-5, we discussed the need to improve runtime error detection algorithm in

shell script and along with that, add logic to add logical errors in the getFeedback and

updateFeedback files is important.

Lastly, one area of interest is to investigate git difference of previous and current versions

to generate formative feedback or any important results that can be used towards automating the

feedback-generation process. We like to believe that there’s some potential work that can lead us

to fruitful results.

34

Chapter 6

Conclusion

Providing formative feedback is important to novice programmers, since a Boolean

result, if the test has passed or not doesn’t help them in resolving their bugs. Also, the availability

of feedback is important as well, since waiting on a TA/instructor for feedback is time

consuming. So, this study proposes an implementation of feedback generation for programming

assignments using continuous integration (CI) tool called Jenkins CI, via crowdsourcing. The

proposed model has an ability to provide formative feedback within seconds and also, the model

is capable of improving itself over-time, based on feedback collected from students. The idea

behind this model was if a student stumbles upon a problem, then use their experience (i.e.

feedback in this case) to help other students overcome the same problem. Also, the proposed

model can be adapted for various programming languages, that can be compiled/ran from shell

script, and is not limited to Java. Finally, it is important that the complete solution is not provided

to students as a feedback, and in instead, partial solution helps them in learning process. Hence, it

was shown in this study, that the proposed model supports and provides partial feedback, in terms

of hints and not complete solutions.

35

Bibliography

[1] Jenkins CI. [Online]. Available: https://www.jenkins.io.

[2] Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A Systematic Literature Review of

Automated Feedback Generation for Programming Exercises. ACMTrans. Comput. Educ. 19, 1,

Article 3 (September 2018), 43 pages.

[3] Susanne Narciss. 2008. Feedback strategies for interactive learning tasks. Handbook of

Research on Educational Communications and Technology. Routledge, 125–144.

[4] Fadi P. Deek and James A. McHugh. 1998. A survey and critical analysis of tools for learning

programming. Comput.Sci.Edu. 8, 2 (1998), 130–178.

[5] Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R. Klemmer. 2010. What would

other programmers do: suggesting solutions to error messages. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems (CHI '10). Association for Computing

Machinery, New York, NY, USA, 1019–1028. DOI:https://doi.org/10.1145/1753326.1753478

[6] Peter Naur. 1964. Automatic grading of student’s ALGOL programming. BIT Numer. Math.

4, 3 (1964), 177–188

[7] Valerie J. Shute. 2008. Focus on formative feedback. Rev. Edu. Res. 78, 1 (2008), 153–189.

[8] Safei, Suhailan & Shibghatullah, A. & Burhanuddin, Mohd. (2014). A perspective of

Automated programming error feedback Approaches in problem solving exercises. Journal of

Theoretical and Applied Information Technology. 70. 121-129. [9] M. M. Ben-ari, “Compile and

Runtime Errors in Java,” Israel, 2007.

[9] Socure Inc. [Online]. Available: https://www.socure.com

[10] W. Lewis Johnson and Elliot Soloway. 1985. PROUST: Knowledge-based program

understanding. IEEE Trans. Softw. Eng. 11, 3 (1985), 267–275.

36

[11] E. R. Sykes and F. Franek, “An Intelligent Tutoring System Prototype for Learning to

Program Java TM,” in IEEE International Conference on Advanced Learning Technologies

(ICALT 2003), 2003, no. figure 2, p. 7695.

[12] A. Casamayor, A. Amandi, and M. Campo, “Intelligent assistance for teachers in

collaborative e-learning environments,” Computers & Education, vol. 53, no. 4, pp.1147–1154,

2009.

[13] A. Vihavainen, T. Vikberg, M. Luukkainen, and M. Pärtel, “Scaffolding Students’ Learning

using Test My Code,” in Annual Conference on Innovation and Technology in Computer Science

Education, 2013, vol. 68, pp. 117–122.

[14] Stack Overflow. [Online]. Available: https://stackoverflow.com/tour.