Enhancing JFLAP with automata construction problems and automated feedback

Vinay S. Shekhar, Anant Agarwalla, Akshay Agarwal, Nitish B.

Department of Information Science PES Institute of Technology

Bangalore, India

Viraj Kumar Department of Information Science

PES University Bangalore, India

viraj.kumar@pes.edu

Abstract—JFLAP is a powerful and popular educational software tool that allows students in Formal Languages and Automata courses to understand theoretical concepts in a visual and interactive manner. In this paper, we enhance the quality of JFLAP’s interactivity in two ways that are extremely useful for beginners learning to build finite automata: (1) our tool can automatically generate a rich variety of practice problems (and their solutions) and immediately evaluate student answers, and (2) we propose a novel way for students to obtain prompt and meaningful feedback on their solutions to problems taken from other sources such as assignments or textbooks. We believe that these enhancements can greatly improve the utility of JFLAP as an instructional resource.

Keywords—Automatic problem generation and feedback; formal languages and automata; JFLAP

I. INTRODUCTION AND RELATED WORK The study of formal languages and automata (FLA) is a

core component of the undergraduate Computer Science curriculum [1], and a rich variety of software tools have been developed to make the material more engaging and understandable (see [2] for an excellent survey). Amongst these, JFLAP [3] has proven to be extremely popular because it covers nearly every topic in the FLA course (see [4] for world-wide usage statistics). Any extensions developed for this tool therefore have the potential to be widely adopted. Furthermore, it is easy to develop such extensions, and the community actively encourages such enhancements [5]. In this paper, we present enhancements to JFLAP that help beginners tackle automata construction problems. The extensions we have developed are freely available at http://bit.ly/jflapenhanced, with a comprehensive document explaining how to install and use the tool.

Fig. 1. A DFA (drawn in JFLAP) for the example language L*

Automata construction problems are typically introduced very early in the FLA course. Students are given a set of strings L satisfying a certain property (more formally, L is a regular language) and asked to construct a deterministic finite automaton (DFA) that accepts precisely the strings in L. For example, consider the language L* containing all binary strings x such that (i) x contains an even number of 1’s, and (ii) the integer corresponding to string x is divisible by 3. Students attempting to construct a DFA for this problem are faced with two challenges: (1) they must understand the definition clearly enough to identify exactly those strings that belong to L*, and (2) they must design an automaton that accepts exactly these strings. Challenge (1) is a stumbling block for students lacking mathematical understanding, particularly in propositional logic, quantifiers, and the meaning of specific predicates in the English language. The difficulties faced by such students in the FLA course are well documented [6][7], but it is also known that students can improve their automata construction skills with practice [8]. To help students address challenge (1), we equip JFLAP with a large set of practice problems. While tackling a practice language L, students can request a sample of strings in L, and can also query whether a particular string x is L or not. Furthermore, our tool allows students to test their tentative solution and receive immediate feedback in the form of strings that their DFA should/should not accept. This facility in described in detail in Section II.

Even for students with sufficient mathematical skills to begin developing a solution, challenge (2) remains a significant hurdle because JFLAP offers very limited feedback. For example, consider the incorrect DFA for L* in Fig. 2. This accepts all strings in L*, but it also accepts strings outside this language (viz. strings that satisfy only condition (ii) but not condition (i) in the definition of L*).

Fig. 2. An incorrect DFA for the example language L*

978-1-4799-5173-4/14/$31.00 ©2014 IEEE 19

public boolean inLanguage(String x) { int numOnes = 0; for(int i = 0; i < x.length(); i++) { if(x.charAt(i) == '1') numOnes++; } return (numOnes % 2 == 0) && (Integer.parseInt(x, 2) % 3 == 0); }

public boolean inLanguage(String x) { boolean evenOnes = true; int remainder = 3; for(int i = 0; i < x.length(); i++) { if(x.charAt(i) == '1') { evenOnes = !evenOnes; remainder = (2*remainder+1) % 3; } else remainder = (2*remainder) % 3; } return evenOnes && (remainder == 0); }

The only feedback JFLAP presently offers students is the ability to test whether a particular string x is accepted by their DFA or not, and the automaton in Fig. 2 correctly classifies several strings. (This is because most strings that satisfy condition (ii) also satisfy condition (i).) Therefore, a student may fail to find a counter-example despite conducting several tests and (erroneously) conclude that this DFA is a correct solution. On the other hand, an instructor (who knows the correct solution) can use the well-known DFA equivalence algorithm already implemented in JFLAP to detect that the two automata are in fact inequivalent. This fact can later be shared with students, but this kind of feedback is neither instantaneous nor useful in helping students understand their errors.

A simple extension to JFLAP [9] partially addresses the need to provide students more detailed feedback: if the student’s DFA S is inequivalent to the instructor’s solution T, the tool reports a shortest counter-example x* that is accepted by exactly one of the DFAs S and T (as per Occam’s principle, short counter-examples are likely to be more informative than longer ones). For our hypothetical example above, the shortest counter-example is x* = 10101 whose binary value (21) is divisible by 3 but which has an odd number of 1’s. This string is accepted by S (Fig. 2) but not by T (Fig. 1). A more sophisticated feedback mechanism has been proposed [10] (although this has not yet been integrated with JFLAP). When S and T are not equivalent, this latter technique can determine (a) the fewest number of changes that need to be made to S to make it equivalent to T (on the assumption that this minimal set of changes captures the “errors” made by the student), and (b) the fewest number of changes that need to be made to the description of strings accepted by S to make it equivalent to the description of strings accepted by T (on the assumption that these differences in description capture the extent to which the student misunderstood the question). This feedback is extremely valuable to instructors for evaluation purposes [10], but once again it is not immediately available to students.

Therefore, in this paper we also present a novel way to give students automated and immediate feedback on automata construction problems that are not part of our practice compendium (these problems could be drawn from other sources – homework assignments and textbooks, for instance). In order to obtain feedback for their DFA S constructed for language L, students write a Java function inLanguage() to determine whether a given input string is in the language L or not. Our tool compares the programmatic solution inLanguage() with the DFA solution S and reports counter-examples, if any. We defer the details to Section III, and confine ourselves here to justifying this approach.

Fig. 3. A programmatic solution to determine membership in L*

The FLA course is normally offered to students after they have taken two or more programming courses. When faced with the task of recognizing strings in a given regular language L, we believe it is quite easy for many students to provide a correct programmatic solution (once they have overcome challenge (1) described earlier). In order to construct a DFA, however, students must get to grips with the somewhat unfamiliar constraints imposed by the automaton model of computation. To better appreciate this point, consider the problem of constructing an automaton for the regular language L* defined earlier.

A programmatic solution to determine whether a given string belongs to this language is straightforward (Fig. 3), but note that this solution performs computation that is forbidden in the DFA model (viz. counting the number of 1’s and calling a sub-routine). It is possible (but more challenging) to implement this function in a manner more consistent with the DFA model (Fig. 4), but the key point here is that there is no need for the student to write such code. The more natural solution to the problem of membership in L* presented in Fig. 3 is sufficient for us to compare it against the student’s DFA and give useful feedback for the latter, as explained in Section III.

The remainder of this paper is organized as follows. In Section II, we explain our methodology for creating practice problems, providing helpful hints, and validating student solutions. Section III demonstrates how the programmatic solution can be used to provide helpful hints to students about their DFA. This paper is focused on developing feasible and potentially useful enhancements of JFLAP for learners, and is based on first-hand experiences (as students) and student observations (as course instructor). A careful evaluation of these enhancements remains to be conducted, and is a key focus area for subsequent work as explained in Section IV.

II. AUTOMATA CONSTRUCTION PROBLEMS When constructing an automaton for a target language L,

the property satisfied by strings in L is usually expressed in a natural language (such as English). This is especially true for introductory problems, although students may later be expected to solve problems where the language L is defined in other ways, for instance as a regular expression. Accordingly, we have enhanced JFLAP with two kinds of practice problems as show in TABLE I.

Fig. 4. A solution for L* more consistent with the DFA computation model

20

TABLE I. TWO SETS OF PRACTICE PROBLEMS

Description of strings in L Generation Number

English Manual 100

Regular expression Automatic Unlimited*

* Generated by creating a random regular expression of specified (but arbitrary) depth d

TABLE II. A FEW COMMON PREDICATES

Predicate Example of usage in problem

Length “…strings whose length is 3…”

Divisible by “…length is divisible by…”

Starts/Ends with “…ends with 001…”

Contains (exactly/at least/at most) “…contains 001…”

For the present work, we have manually created 100 problems similar to those found in reference textbooks [11] (we are currently investigating ways in which even this process can be automated). For creating “similar problems”, we first examined over 500 problems from these sources and created a list of 37 common predicates – a sample is listed in TABLE II. We used these predicates together with logical operators, quantifiers, and basic arithmetic operators to create candidate properties to be satisfied by strings in our languages, such as:

• AND(containsExactly(x, 2, 1), divisibleBy(length(x), 4)) • containsAtLeast(x, length(x)/2, 1)

The former expression corresponds to the regular language of “binary strings containing exactly two 1’s, whose length is divisible by 4”, and problems such as this were retained. Note that the latter expression corresponds to the non-regular language of binary strings where at least half the total number of bits is 1 – such problems were discarded.

In addition to the list of problems we provide, our tool lets users (both instructors and learners) add more problems to this collection by drawing a DFA and clicking “Save As Problem”. As shown in Fig. 5, the user is prompted to enter a problem description in English, which is then added to the repository along with the DFA solution.

Fig. 5. Saving a new problem

Fig. 6. The “Quiz Me” interface for a practice problem L92

Our tool can also automatically generate problems where the target language L is specified as a regular expression. The user is prompted to select the maximum “depth” of the regular expression, and we generate a random regular expression of this depth d. Since there is no pre-defined limit to d, the number of such problems we can generate is effectively limitless.

Both kinds of practice problems are accessible by clicking “Quiz Me” in the Finite Automaton view of JFLAP (Fig. 5). We demonstrate this functionality in Fig. 6 with the practice problem L92 consisting of binary strings where every “1” is followed by “00”. The user is expected to click on “Draw the DFA” to begin attempting the problem.

Since the solution to each practice problem is known, we can provide the learner with a rich variety of hints. For problems described in English (as in Fig. 6), we automatically generate the regular expression corresponding to this language and provide it as a default hint to help students visualize the pattern of strings to be accepted. (JFLAP’s built-in DFA to regular expression converter uses the standard state-elimination algorithm that is taught in most FLA courses. This tends to generate bulky, hard-to-read regular expressions, so we have implemented Brzozowski Algebraic Method [12], which is based on Arden’s Lemma [13] and produces significantly shorter expressions on our practice set of examples.) Learners who are unfamiliar with regular expressions can click on “View Possible Strings” to see a random selection of strings in the language (generated using the Xeger library [14]). The list of strings will be different each time the button is clicked, unless the language itself has very few strings.

Students are sometimes unsure whether a particular string is in the language or not. For instance, is the string “0” in L92? The answer is “yes” because “0” contains no 1’s, and hence it is vacuously true that every 1 in this string is followed by 00. This is precisely the kind of careful mathematical reasoning skills that students are expected to possess after overcoming challenge (1) (see Section I). To help students practice these skills, we provide a “Test Input String” feature that identifies whether a particular user-supplied string is in the language or not. This is not the same as JFLAP’s existing functionality of testing whether a particular string is accepted by the DFA or not: the “Test Input String” feature reports the correct answer even if no attempt has been made to construct a DFA yet.

21

Assumption: If the programmatic solution and the DFA solution are inequivalent, then there is a “short” counter-example establishing this fact.

Fig. 7. Automated feedback for an incorrect DFA

After clicking “Draw the DFA”, the student sees a panel similar to the regular JFLAP interface for drawing DFAs, but with an additional “Check your DFA!” button. Clicking this causes the tool to compare the student DFA with the (internal) DFA solution, and we report either a success message or the existence of counter-example(s). In the latter case, we present several counter-examples when they exist (in [9], only a single counter-example is presented), and we further label these as “incorrectly accepted” or “incorrectly rejected” by the students’ DFA. If a student is unable to solve the problem, clicking “Show me the Answer” (Fig. 6) reveals the solution.

III. AUTOMATED FEEDBACK There are well-known efficient algorithms for determining

the equivalence of two DFAs, but since we impose no constraints on the code within the inLanguage() function, determining whether or not the programmatic solution is equivalent to the DFA is undecidable [15]. As a practical heuristic, however, we make the following assumption which we believe is reasonable for the type of problems encountered in the FLA course:

In view of the above assumption, we simply examine all possible strings up to a maximum string length k chosen by the user based on the time available. To illustrate the utility of this simple approach, consider an incorrect DFA and the associated inLanguage() function for the language L92 in Fig. 7. The DFA has several errors. Firstly, the initial state q0 must be an accepting state: the empty string is in the language because it has no 1’s. The student gets a “Mismatch” warning for this input, and can investigate the error. Secondly, the DFA does not define any transition from the initial state on input 0 (there is no arrow labeled 0 from q0). JFLAP interprets this missing transition as an implicit transition to a rejecting state from which the DFA cannot escape (a “trap state”). This oversight is conveyed to the student by “Mismatch” warnings on several inputs that begin with “0”.

It is impossible to know whether “mismatches” are due to faults in the DFA, or in the inLanguage() function (or, indeed, both). We expect that since programming syntax and semantics are more familiar for students, they will find it easier to debug the programmatic solution, and attribute remaining errors (if any) to the DFA. The inLanguage() function in Fig. 7, for example, contains a logical flaw which causes an “index out of bounds” exception on the string “1”, and on any other input that ends in “01”. Our tool handles such exceptions gracefully: the corresponding inputs are appropriately flagged for students to address.

Exceptions may also arise due to ambiguities in the given problem description. For instance, the code in Fig. 3 raises an exception on the empty string input (parseInt() requires the string argument to be non-empty). The language L* consists of binary strings with an even number of 1’s (which is true for the empty string) and whose corresponding integer value is divisible by 3. Since there is no natural definition of the integer corresponding to the empty string, one can argue that there is ambiguity here. Our tool can therefore be used by instructors to detect such pitfalls ahead of time and formulate questions that are more robust. In the case of L*, for instance, the instructor could specify that the language consists of non-empty strings x satisfying the given conditions (i) and (ii).

We note that it is impossible for students to conclude from a lack of errors that their DFA is correct. To begin with, the choice of the maximum input length k may have been smaller than the length of the shortest counter-example. Another concern is that the DFA could be replicating errors in the programmatic solution, both of which stemmed from a more fundamental misunderstanding of the problem. We therefore stress that our automated feedback mechanism may be unhelpful to students who are yet to overcome challenge (1) (see Section I). This facility also cannot help students who lack the programming skills necessary to translate the language membership problem into an appropriate inLanguage() function.

For a majority of students, however, we believe that this extension to JFLAP is extremely useful, especially because the feedback it creates is self-generated (i.e., there is no reliance on the course instructor, teaching assistant, etc. to provide help). Thus it is immediate, and personalized to the student’s own approach. It can also scale well to large classes (including very large online classes i.e., MOOCs) where personalized manual feedback is impractical.

Our approach requires students to perform an additional programming task, which may be unpalatable for some. If instructors believe that this effort is counterbalanced by improved learning, they can include the programmatic solution as a part of the submission for every automaton construction assignment. Our tool can, of course, automatically evaluate students’ programmatic solutions by comparing them against the instructor’s DFA solution.

Lastly, we note that our tool allows instructors to define a new type of automaton problem: given an instructor-defined DFA, students must describe its language as a programmatic solution (using high-level programming constructs).

22

Fig. 8. A new type of problem: describe a DFA using high-level

programming constructs (incorrect solution)

An example of such a problem is shown in Fig. 8. One can trivially translate any DFA into a programmatic solution that mimics the state transitions, but the challenge is to characterize the DFA behavior at a higher level of abstraction. The DFA in Fig. 8 determines whether a given binary string x contains the pattern “110” followed by at least one more letter. The Java code below the DFA attempts to characterize this language as binary strings x that contain “110” but do not end with that pattern, which is almost (but not quite) the same. Our tool can again provide students with immediate and useful feedback (in this case, the counter-example “110110”).

IV. FUTURE WORK We have demonstrated that JFLAP’s capabilities can be

enhanced in several ways that benefit learners, but it remains to be tested whether our extensions produce measurable benefits in short-term and long-term understanding. We are in the process of designing such a study (to be conducted in the forthcoming academic year), but we believe that our approach has merits because it parallels a successful strategy in a closely related domain: software debugging. Test-driven development (TDD), made popular by extreme programming [16], is an effective strategy for increasing the quality of student code, with 45% fewer defects per 1,000 lines of code according to one well-cited study [17]. Based on this evidence, the author advocates that “a student should be given the responsibility of demonstrating the correctness of his or her own code”, and recommends using “automated grading and feedback generation to provide for frequent, quick turnaround assessments of student performance”. Our approach, we believe, embodies this philosophy very well.

In addition to constructing DFAs, the FLA course asks students to construct other kinds of state machines and grammars, all of which are well supported by JFLAP. Many of our ideas can be extended quite easily in these directions.

AUTHOR CONTRIBUTIONS The overall project was conceived by V.K. who supervised

the work undertaken by N.B. working on automated feedback using programmatic solutions, and initially supervised the work undertaken by the remaining authors on generating automata construction problems from regular expressions (the latter authors later suggested and implemented automata construction problems from English descriptions based on textbook-style problems). V.K. was the lead author of this paper.

REFERENCES [1] Joint Task Force on Computing Curricula, Association for Computing

Machinery (ACM) and IEEE Computer Society, “Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science”, ACM, New York, NY, 2013.

[2] P. Chakraborty, P.C. Saxena, and C. P. Katti, “Fifty years of automata simulation: a review”, ACM Inroads, Volume 2, Issue 4, p. 59-70, 2011.

[3] S. H. Rodger and T. W. Finley, “JFLAP – An Interactive Formal Languages and Automata Package”, Jones and Bartlett, Sudbury, MA, 2006.

[4] JFLAP website: http://www.jflap.org [5] S. H. Roger, J. Lim, and S. Reading, “Increasing interaction and support

in the formal languages and automata theory course”, Innovation and Technology in Computer Science Education (ITiCSE 2007), pp. 58-62, 2007.

[6] N. Pillay, “Learning Difficulties Experienced by Students in a Course on Formal Languages and Automata Theory”, SIGCSE Bulletin, vol. 41, no. 4, pp. 48-52, New York, NY, 2010.

[7] T. Smith, and R. McCartney, “Mathematization in teaching pumping lemmas”, Frontiers in Education 2013, pp. 1671-1677, 2013.

[8] M. T. Grinder, “A Preliminary Empirical Evaluation of the Effectiveness of a Finite State Automaton Animator”, SIGCSE Bulletin, vol. 35, no. 1, pp. 157-161, 2003.

[9] D. A. Norton, Algorithms for testing equivalence of finite automata, with a grading tool for JFLAP. Masters Project, Department of Computer Science, Rochester Institute of Technology, 2009. URL: https://ritdml.rit.edu/handle/1850/8712

[10] R. Alur, L. D'Antoni, S. Gulwani, D. Kini, and M. Viswanathan, “Automated Grading of DFA Constructions”, Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence (IJCAI'13), pp. 1976-1982, 2013.

[11] Wikipedia article “Theory of Computation” (accessed April 12, 2014): http://en.wikipedia.org/wiki/Theory_of_computation#Further_reading

[12] J. A. Brzozowski, “Derivatives of regular expressions”, J. ACM, vol. 11, no. 4, pp. 481-494, 1964.

[13] D. N. Arden, “Delayed logic and finite state machines”, Theory of Computing Machine Design, pp. 1-35, University of Michigan Press, Ann Arbor (1960).

[14] Xeger library: https://code.google.com/p/xeger [15] John E. Hopcroft and Jeffrey D. Ullman, “Introduction to Automata

Theory, Languages, and Computation”, Addison-Wesley Publishing, Reading, Massachusetts, 1979.

[16] K. Beck, “Test-Driven Development: By Example”, Addison-Wesley, Boston, Massachusetts, 2003.

[17] S. H. Edwards, “Using Software Testing to Move Students from Trial-and-error to Reflection-in-action”, SIGCSE Bulletin, vol. 36, no. 1, pp. 26-30, 2004.

23