Students’ Understanding of Computational Problem-Solving Tasks

Magana, A.J.,* Santiago-Roman, A.,^ Santiago-Santiago, N.,^ Aceros, C.,^ Marshall, B.,* and Matson, E.*

* Computer and Information Technology at Purdue University ^ University of Puerto Rico at Mayaguez

Introduction

Computational thinking embodies multiple reasoning processes or ways of thinking which transcend time and disciplines. As a result, computational thinking represents a skill-set that is changing the way we think by providing an extension to our cognitive faculties1 . These changes are impacting science, technology, and society “on ways new discoveries will be made, innovation will occur, and cultures will evolve” 2 . Because computational thinking constitutes a powerful and pervasive capability to solve problems, researchers have pointed out that there is a need to start teaching computational thinking early and often 4,3

; however several challenges

have been identified 5 . Questions such as: What are the effective ways of learning computational thinking? What are the effective ways of teaching computational thinking? What are the core concepts of computational thinking? How should these concepts be integrated into the curriculum? These are some of the questions that need to be dealt with. To take initial steps toward identifying how students understand a computational problem solving task, we explored how technology and engineering students approach solving computational problems. This study is guided under the premise that people’s ways of experiencing phenomena result from the unique interaction of their understanding of the phenomena and the situation in which they must apply that understanding 6 . Our ultimate goal is that by identifying, comparing, and contrasting students’ understandings and strategies to approach problems, more effective instructional approaches will be identified.

Theoretical Frameworks

Problem-based learning and Computational thinking theoretical frameworks are merged in the design and analysis of this study. Problem-Based learning is a student-centered instructional strategy to guide the learner(s) through a set of steps allowing them to explore and understand the problem, generate potential solutions, investigate them and come up with the most effective solution. Research on problem-based learning suggests this instructional strategy supports students’ learning processes better than traditional instruction 7 . The second strand that guides this investigation relates to Computational Thinking. Computational Thinking is a type of analytical thinking that employs mathematical and engineering thinking to understand and solve complex problems within the constraints of the real world 5 . Computational Thinking possesses unique characteristics and is encompassed with recurring themes. The most important characteristics are: a) it requires thinking at multiple levels of abstraction, b) it represents a fundamental skill to function in modern society, c) it embodies a way humans solve problems, d) it complements and combines mathematical and engineering thinking and e) it uses computational concepts (not computers) to solve problems 8,4 .

The implications of these theoretical frameworks relate to the design and study of students’ understanding and representation of a problem that need to be solved by means of computational thinking. Although problem-based learning involves a collaborative iterative process of understanding the problem, analyzing it, and finding the solution, in this study we focused our efforts in identifying students’ individual understanding and representation of the problem. In particular, what we tried to identify was a) students’ level of abstraction of the problem and b) students’ representation of the problem. The specific research question for this study is: How students understand, represent, and respond to a computational problem–solving task?

Methodological Framework

Qualitative research was the inquiry method employed for this study. Qualitative research methods have been applied for the “study [of] things in their natural settings, attempting to make sense of, or interpret, phenomena in terms of meanings people bring to them” (p.3) 9 . That is, qualitative research methods focus on the investigation of what phenomena means to the participants, how it affects them, how they think about it, and what they do about it 10 . For this particular study, the purpose of investigating students’ qualitative understandings of the problems (through qualitative research methods) provided the researchers a way to explain students’ ways of understanding a computational task. The methodological framework that guided this study was Grounded Theory11 . Grounded Theory is a research method that operates almost in a reverse fashion from traditional research. Rather than beginning by researching and developing a hypothesis, the first step is data collection. From the data collected, the key points are marked with a series of codes extracted from the text. The codes are grouped into similar concepts in order to make them more workable. From these concepts, categories are formed, which are the basis for the creation of a theory, or a reverse engineered hypothesis 12 .

Methods

Participants and procedures. The participants for this study were 19 sophomore-level technology students (T) from computer and information technology at a Midwestern University (MU) and 16 sophomore-level engineering students (E) from a five year computer engineering program at a Hispanic Serving Institution (HSI). While the students in the MU belong to the College of Technology where computing is taught in an application oriented environment, the students from the HSI belong to the College of Engineering where design and processes are highly emphasized. Technology students were enrolled in an object-oriented programming class where in addition to learning object-oriented concepts and ideas (i.e. encapsulation, inheritance, aggregation, etc.,) they also learned about data structures. Engineering students were enrolled in a data structures course. Also, both groups were exposed to the same intervention where students had 40 minutes to work on the case study and the solution. The researchers only answered clarifying questions about the process. Also, students were asked to focus on the process required to solve the problem, not on the solution. Data collection method. The data collection method was a case study followed by an ad-hoc assessment. The case study was adapted from Dale13 and consisted of a problem-solving task where students had to design a simulator. The task consisted of creating a simulation that determined how long items (people, jobs, cars …) must wait in a single line before being served.

The simulation varies the number of servers and the time between arrivals of the items. The assessment consisted of asking students to describe: a) what is the problem they have been asked to solve, b) all the elements to be considered that should be modeled into the simulator (e.g. inputs and outputs, rules, etc.), and c) how they go about solving this task and d) their solution by means of a model. Case problem. Students were presented with a traditional supermarket scenario to generate a simulation model that determines how long customers must wait in a single line before being served. The following systems characteristics were to be met: (1) the number of cashiers required to be kept within the limits set by the store’s budget, and (2) the average customer required not to be kept waiting too long. Also, they were provided with the following guiding question: how does the company determine the optimal compromise between the number of servers and the waiting time? Finally, students had to follow four steps: (1) describe the problem you have been asked to solve; (2) describe all the elements to be considered that should be modeled into the simulator (e.g., number of servers); (3) describe how you would go about solving this task and describe why you made each decision; (4) describe your potential solution using a model (or models). Appendix A presents the problem under study. Data analysis method. The data analysis focused on identifying similarities and differences between students’ understandings of the problem and their different approaches to solve it. Data analysis was conducted following a two-step process. The first step consisted of analyzing the set of responses from the 19 technology students and then, the second step consisted of analyzing the set of the 16 engineering students. First, two researchers coded the first set of data independently and then discussed the findings. Whenever researchers were in disagreement, an in-depth discussion followed until an agreement was met. Then, data was re-coded based on the newly agreed procedures. Students’ responses were analyzed using open coding. While conducting open coding, the researchers followed the following steps: (1) numbered students’ solutions from 1 to 19 to be able to identify them, (2) grouped students’ answers based on their understanding of the problem, (3) identified the elements of the system considered to generate a solution, (4) analyzed students’ description of the potential solution and (5) identified students’ graphical representations of the problem. For each step, the researchers identified similarities and differences on students’ understanding of the problem presented. Results were compared and analyzed between the researchers until 100% agreement was reached. The second step was to conduct the same analysis for the set of responses corresponding to the 16 engineering students. One of the researchers who participated in the first data analysis also conducted the same data analysis for the second set. Two more researchers participated in the analysis of the second data set and followed exactly the same procedure conducted for the first data set. At the end, the results of both data analyses were compared.

Results

Students’ understanding of the problem. The researchers analyzed students’ responses and identified three main categories of understanding the problem: (1) waiting time, (2) optimization, and (3) element identification. Two engineering students, E8 and E15, did not address the problem; thus, they were not included in the analysis. Table 1 summarizes students’ distribution for each category. Technology and engineering students were identified as T# or E# respectively, where # represented the identifiers assigned to each student.

Table 1. Distribution of students according to the solutions generated.

Category

Name Description Participants Total %

1 Waiting

Time

The student identified that the objective of the problem was to

determine the waiting time.

T1, T3, T5, T9, T10, T11, T12, T13, T15, T17

10 53

E1, E5, E6, E10, E14 5 33

2 Optimization The student visualizes the problem

as an optimization of a cost function.

T4, T7, T16, T19 4 21 E2, E3, E4, E11, E12,

E13, E16 7 47

3 Element

Identification

The student perceived the primary objective of the problem was to

identify the elements of the system.

T2, T6, T8, T14, T18 5 26

E7, E9, E17 3 20

Students who focused their solution around waiting time interpreted the problem as being asked to (1) “determine the average waiting time to be served in a single line” (T15) or (2) to “design a system to calculate what the time a person needed to wait before being served” (T17) or (3) “how long someone must wait in a single line before being served” (E5). Students who focused their solution on optimization interpreted the problem as to (1) “optimize the checkout process of a supermarket” (T7) or (2) “determine the optimal solution for moving the maximum amount of items through a line with minimum amount of servers” (T4) or (3) “determine the number of servers that I need to optimize the service and the cost” (E3). Students who focused their solution in element identification interpreted the problem as to (1) “develop a simulation so a store can determine how many cashiers are needed” (T2), or (2) find out how many servers and cashiers are needed” (T6), or (3) “simulate amount of servers serving or attending different items with varying times” (E9). Identified elements of the system. The researchers identified the elements students’ deemed necessary to solve the problem according to their written responses. A total of 20 different elements were identified among the 35 students from both institutions. The top five elements identified by both groups and the corresponding description for each are presented in Table 2. Overall, three elements, the number of servers, the number of clients to be served and the waiting time, were identified by over 50% of the total sample. A sample of students’ responses identifying number of cashiers was: “number of servers” (E1). A sample of students’ responses identifying number of customers was: “clients”/“number of clients”. And a sample of students’ responses identifying the waiting time was: “time a customer should wait” (E6). Students’ descriptions of the potential solution. The researchers also evaluated students’ description of the potential solutions considering: (1) the category in which they were originally classified (i.e. waiting time, optimization, and element identification) and (2) the elements considered by the student in the solution of the problem (e.g. number of cashiers, customers, waiting time, etc.). These data were obtained from the analysis done in steps 1 and 2 respectively. In their explanations, the technology students focused on six main areas: (a) input,

(b) output, (c) rules, (d) equations, (e) implementation, and (f) optimization. In addition, engineering students had an additional area in their explanations: data structures (see Table 4). Table 2. Top five elements considered by both groups of participants.

Elements Description Percentage of participants

Technology Rank Engineering Rank

Number of servers Number of servers that are available to the

clients to complete their transaction. 95% 1 80% 1

Number of clients to be served

Number of clients waiting or being served. 89% 2 67% 3

Waiting time Amount of time elapsed between the arrival to the line and the completion of the service

from the customers’ perspective. 63% 3 20% 6

Time constraints Any time concern not related to waiting time. 42% 4 80% 1

Server features Characteristics of the server used to process

the transaction. 37% 5 13% 8

Number of items Number of items to be purchased. 32% 6 67% 3

Budget Available amount of money for the system. 5% 10 27% 5

Table 3. Students’ description of the potential solution.

Area of analysis

Description Category 1 Category 2 Category 3 Total

Input Inputs to the solution of

the system.

T1, T9, T10, T11, T12, T13, T15, T17

T4, T7, T19 T2, T6, T14,

T18 79%

E1, E5, E10, E14 E3, E4, E11, E13, E16 E17 67%

Output Output of the solution

of the system.

T3, T5, T9, T10, T12, T13, T15, T17

T7, T16 T6, T8 63%

E1, E10, E14 E4, E12, E13, E16 E7, E17 60%

Rules Compliances/restriction

s established by the student.

T3, T12, T15, T17 T7, T16, T19 T2, T6, T8,

T18 58%

E1, E5, E6, E10, E14 E2, E3, E12, E13 E7, E9 73%

Equation A mathematical

relationship established by the student.

T10, T12, T13, T17 T16, T19 T6 37%

E1, E14 E2, E12 E9 33%

Implementation

The particular way and specifics of the

implementation of the solution,

T3, T5, T9, T10 None None 21%

E10, E14 E4, E11, E12, E16 E17 47%

Optimization Description of variables to be optimized within

the cost function.

T3, T9 T7, T16 T6, T8 32%

E10 None E17 13%

Data structures included

Data structure to be used in a particular

solution for the problem.

None None None 0%

E1, E6, E10, E14 E3, E4, E13, E16 E7, E9, E17 73%

The first three areas (input, output, and rules) focused primarily on the problem analysis or description. For example, response from participant T1 was coded as input because he/she focused only on the things needed for the system. The student stated the following: “I would mentally go through all of the aspects involved in the system such as items, servers, customers, etc., to get a better idea of what the system needs to do…” Another example is the response from student E10 who mentioned “The input would be the amount (or list) of merchandise the customer has, and the time it arrives to the waiting line, and the amount of servers”. Very few participants considered all aspects (input, output, and rules), explicitly in their explanations. Participant T15 is one case in which he/she clearly stated the input: “customers would be input to the shortest server queue”; rules: “items would be delivered to the least customers in each queue at random intervals”; and output: “the total time in a queue is logged and averages are computed for varying server levels”. Student E5 provided a similar description of a rule: “I will create a program that will determine if that server has less people than others and the time it takes is also less that the others will give the option to take more people from other servers so we can have the same time at each server.” The fourth area, equation, was assigned to responses including an explanation, either graphical or verbal, of an equation to be solved. For instance, participant T10 explained “the amount of time a server works minus the amount of time between time arrivals for that sever all divided by the ArrayList item count for that server will equal the amount of time each item has to wait in a particular server line”. In the same category, equation, participant E14 responded “total waiting time = (# of items in queue * (time spent by worker on an item + time an item takes to reach the worker)) / (# of workers)”. Similarly, participant T19 used a Graphical Model explanation which is presented in Figure 3. Finally, implementation and optimization was assigned to responses related to the application of the designed solution. An example of both implementation and optimization was provided by participant T10 who explained his /her solution as “First I would receive each of the servers through user input, and each time a new Item is input it will be added to an ArrayList named for that particular server. I will then count the items in a server’s ArrayList…”. A second example of implementation would be: “Since we have to take in mind that it's a single line, it's best to use a list structure, and since a waiting line has the “first in first out” characteristic, a queue would be most effective...” (E10). An example of optimization would be: “I would calculate the ideal number of registers open for each time period that would minimize customer wait time and stay under the allotted budget” (T8). Students’ representations of the potential solution. Students were asked to draw a model representing their solution to the problem. More than half of the students (over 60%) came up with a potential graphical representation for a solution to the problem. Table 4 summarizes the kinds of representations students came up with together with the percentages. Sixteen percent of the technology students modeled a process, as shown in Figure 1, while none of the engineering students applied this method. Other technology and engineering students used a mathematical equation as the example shown in Figure 2. Also, a few technology and engineering students modeled their solution as a graphical user interface (see Figure 4), and a few more sketched a physical layout (see Figure 3). In addition, one technology student made a class diagram shown in Figure 5 and an engineering student represented his/her solution with pseudo code (Figure 6). Finally, a couple of engineering students represented their solution through a flowchart (refer to

Figure 7) and few other engineering students represented their solution through a data structure diagram as shown in Figure 8. Table 4. Students’ representation of the potential solution.

Model Description Category 1 Category 2 Category 3 Total

No model Representations were not found in

students’ responses. T1, T11, T15 None T6, T14, T18 32%

E5, E6 E2, E4, E13 E7 40%

Process Model of the data movement

process in the system. T5, T10 T7 None 16%

None None None 0%

Mathematical Description where a mathematical

equation or a relationship is established by the student.

T12 T16, T19 None 16%

E1, E14 E12 None 20%

Layout Representation of the placement of

the stations in the store. T3, T17 None T2 16%

E1, E14 None None 7%

Graphical interface

Prototype of the look and feel of the user interface.

T9 T4 None 10%

None None None 0%

Class diagram

Diagram of the classes used in the object-oriented model and their

relationships. It also contains the attributes of the classes.

None None T8 5%

None None None 0%

Code Pseudo code or actual code used to

solve the problem. None None None 0%

E1 None None 7%

Data structure Graphical representation of how the data is saved in the system.

T13 None None 5%

E1, E10 E3 E9, E17 33%

Flowchart Temporal graphical representation

of the phases traversed by the code.

None None None 0%

None E3, E16 None 13%

Figure 1 - Model of a process (T5)

Figure 2 - Mathematical model (T19 & E14)

Figure 3 - Model of a physical layout (T2)

Figure 4 - Graphical user interface model (T9)

Figure 5 - Model of a class diagram (T8)

Figure 6 - Pseudo code Model (E1)

Figure 7 - Flowchart model (E3)

Figure 8 - Data Structures model (E17)

Discussion and Implications

Findings from this exploratory study shed some light into students’ level of understanding of problems, the factors and variables they considered for solving those problems, representation of such problems, and potential solutions. Here, we have identified how two very different groups of students (i.e. engineering and technology) with different ethnic backgrounds (i.e. Hispanic and non-Hispanic) understood and approached a computational problem solving task by means of abstraction and representation of the problem. Similarities found were in the way students described the problem to be solved (i.e. waiting time, optimization, element identification) and the elements to be considered for solving the problem (i.e. inputs, outputs, etc.). Differences found relate to the variation on students’ responses regarding their descriptions of the potential solutions and the way they chose to represent the problem. For instance, from the students’ understanding of a problem, it was identified that some of them re-stated the problem to be solved, some others went beyond just solving the problem and were looking for the way to optimize the system, and others only listed the elements to be considered to solve the problem. Students were able to understand the problem in very similar ways represented in the three main categories of waiting time, optimization, and element identification. However, it can be identified that the majority of the technology students understood the problem as the task of computing the waiting time, while almost half of the engineering students considered it as an optimization problem. Comparing the frequency rankings of student-mentioned elements, both, technology and engineering students, were able to identity the same and most important elements to be considered in solving the problem. However, while most of them were able to abstract the most important details of the problem from the ones to be ignored, some other students went into a level of detail that perhaps was not relevant to the problem (e.g. discounts, size of the product, store location, etc.).

Considering the potential solutions of the problem, some of the students’ potential solutions were very superficial by focusing only on the elements to be considered to solve the problem (i.e. inputs, outputs). A few students focused on how all required elements to solve the problem should be combined appropriately into a rule (T7, T6, T12, T15, T17 and E1, E10, E13, E14) or an equation (T6, T10, T12, T13, T17, and E1, E2, E9, E12, E14). It was also identified that some of the participants, mainly engineering students, focused more on aspects of implementation rather than describing their strategy to approach the solution of the problem. This difference may be attributed to the effect of the context in which students participated in the study. That is, engineering students may have understood the problem in the context of their data structure class. Finally, when considering students’ representations of their potential solution, it was identified that approximately 30% of the technology students and 40% of the engineering students did not use any model to represent their solution. The rest of the representations ranged in different aspects of the problem including representing the physical layout, a process, mathematical model or some aspect of the programming of the solution. Computational Thinking encompasses a unique combination of reasoning processes and problem solving skills that involve the problem formulation by means of abstraction and decomposition transforming it into one students’ may know how to solve. An important outcome of a reformulated project is an appropriate representation—a model—to convey suitable information by means of reduction, embedding, transformation, or simulation 14,4 . From this study we can identify that students’ understanding of a problem and its potential solution may require a good representation of the problem. However, some students’ representations, when present, were very vague and some others were very superficial. The educational implications of this study can be divided into pedagogical implications and assessment implications. This study reveals the need of a strong emphasis from instructors of having students go through a detailed process of analyzing the problem and come up with an appropriate representation of a problem before moving on into the next stage—the solution of the problem. A potential pedagogical approach that could guide students through this process could be the explicit implementation of problem-based learning as a pedagogical framework to guide students in their process of tackling computational problems. In that form, students may go through a systematic process of analyzing the problem before jumping right into the solution. For example, students may start by formulating or analyzing the problem (describing the problem), identify relevant facts or elements from the scenario (e.g. input, output, rules, etc.), represent the problem (by means of text or diagram), and identify potential solutions to a problem. On the other hand, assessment implications of this study can inform the development of a rubric to guide the assessment of computational problem solving tasks, such as the four step process that has been employed to analyze the data of this study. Identifying how students understand a given problem, which and how many variables they can discriminate, how they graphically represent the problem, and how they brainstorm or troubleshoot potential solutions, can be used as performance outcomes to be measured in a problem solving task.

Conclusion

Computational thinking represents an attitude and skill-set encompassing a unique combination of reasoning processes and problem solving skills. Computational thinking involves formulating a problem by means of a process of abstraction and decomposition transforming it into one students may know how to solve. On the other hand, problem-based learning is a student-centered instructional strategy that guides the learner(s) into a set of steps that allow them to explore and understand the problem, generate potential solutions and investigate them, and come up with the most effective solution. In this study we have combined these two theoretical strands to investigate how students understand, represent, and respond to a computational problem–solving task. We propose that these two strands be used as a pedagogical framework capable of guiding students through a problem understanding and problem solving task. Findings from this study reveal that not all students were able to clearly identify the problem to be solved. Some students did not show an appropriate level of abstraction lacking the ability to identify the most relevant details related to the problem and ignoring others. Furthermore, some students did not come up with a solution at all. Finally, another group of students did not use graphical representations as a tool to aid them to represent the solution to the problem. Although this study did not fully integrate computational thinking with problem-based learning (i.e. problem-based learning was not conducted into the final stage and it was not done collaboratively), we suggest that integrating these two approaches as a pedagogical method can serve as a scaffolding framework for guiding students through the solution of a computational problem-solving task. Further studies will be conducted to test the efficacy of the proposed framework to not only guide students through their learning process, but also as a framework that can help instructors evaluate how students approach computational problem solving tasks.

References:

1. Henderson, P. B., Cortina, T. J., & Wing, J. M. (2007). Computational thinking. ACM SIGCSE Bulletin, 39(1), 195--196.

2. Wing, J. M. (2009). Computational Thinking. Paper presented at the CCSC Northeastern Conference. SUNY Plattsburgh, Plattsburgh, NY

3. Fletcher, G. H. L., & Lu, J. J. (2009). Human computing skills: rethinking the K-12 experience. Communications of the ACM - Inspiring Women in Computing 52 (2).

4. Lu, J. J., & Fletcher, G. H. L. (2009). Thinking about computational thinking. SIGCSE '09 Proceedings of the 40th ACM technical symposium on Computer science education 41 (1).

5. Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions A, 366 (1881), 3717-3725.

6. Bowden, J. and Walsh, E. (2000). Phenomenography. Qualitative Research Methods series. RMIT University Press. Melbourne

7. Hmelo-Silver, C. (2004). Problem-based learning: what and how do students learn? Educational Psychology Review, 16(3), 235-266.

8. Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(2), 33-35.

9. Denzin, N. K., & Lincoln, Y. S. (2000). The discipline and practice of qualitative research. Handbook of qualitative research, 2, 1-28.

10. Patton, M. Q. (2002). Qualitative research & Evaluation Methods. Thousand Oaks, Ca.: Sage Publications.

11. Charmaz, K. (2006). Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis: Sage.

12. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL Aldine.

13. Dale, N. (2007). C++ Plus Data Structures (Fourth ed.). Missiissauga, Ontario: Jones and Bartlett Publishers.

14. Denning, P. (2009). Beyond computational thinking. Commun. ACM, 52(6), 28--30.

Appendix A – Problem studied

To put this situation on a personal level, start with this fact: No one likes to stand on line. If there were one checkout counter for each customer in a supermarket, the customers would be delighted. The supermarket, however, would not stay in business very long. Consequently, a compromise is made: The number of cashiers is kept within the limits set by the store’s budget, and the average customer is not kept waiting too long.

How does the company determine the optimal compromise between the number of servers and the wait time? One way is by experience; the company tries out different numbers of servers and sees how things work out. This approach has two problems: it takes too long and it is too expensive. Another way of examining this problem is by using a computer simulation.

Your task consists of creating a simulation that determines how long items (people, jobs, cars …) must wait in a single line before being served. The simulation must vary the number of servers and the time between arrivals of the items.

Step 1: Please, using your own words, describe what is the problem you have been asked to solve.

Step 2: Please describe all the elements to be considered that should be modeled into the simulator (e.g., number of servers).

Step 3: Please thoroughly describe how you would go about solving this task (your strategy for finding the solution to this problem). Also, please describe why you made each decision. Examples of things you can consider in your solution are:

- Inputs and outputs of the simulator

- What would be the rules of the simulator

- Type of data structure to be used (if required)

Step 4: Describe your potential solution using a model (or models).

Note: It is not required from you to implement the solution, nor to write the code in paper. What we are interested in is on how you understand the problem and how would you approach solving the task.