Analysis of Learning Behavior Logs based on a Fill-in ...€¦ · The application of learning analytics is attracting increasing attention as one possible approach to helping solve

Abstract—It is difficult for teachers to assess overall class

performance in lecture situations where there are more than 100

students per class. In such situations, it is difficult to decide when to

take appropriate remedial action. In an attempt to assess the overall

class attitude and degree of understanding based on the learning

behavior shown by all the students in a large class, including

inattentive or inactive students, we analyzed data logs generated by a

fill-in workbook system developed by the authors. This time series

data made it possible to identify whether students were effectively

following the class learning schedule. Analysis of the results

confirmed that the degree of understanding shown by individual

students could also be usefully assessed by employing this approach.

Keywords—Lecture support, Filling in blanks, Real-time learning

analytics, Learning behavior log, Learning attitude, Degree of

understanding

I. INTRODUCTION

EACHING faculties need to keep an eye on each individual

student's performance during class and to flexibly adjust

class progress according to each student's attitude, mental

condition (i.e., concentrating, being sleepy, unable to follow

course content, giving up, etc.) and degree of understanding.

For example, if the number of students who have lost

concentration increases, faculty members could then let students

discuss the topic with each other or work on exercises instead.

However, it is difficult for teachers to accurately assess the

overall situation in a large class with more than 100 students. As

a result, it is difficult to know when to take appropriate remedial

action in keeping with the current classroom situation.

The application of learning analytics is attracting increasing

attention as one possible approach to helping solve these

problems. Learning analytics is a research field that contributes

to improving learning efficiency by collecting and analyzing

learning behavior logs in e-learning systems. This makes it

possible to predict learners' future learning outcomes and

Kousuke Abe is with Course of Information and Computer Sciences,

Graduate School of Engineering, Kanagawa Institute of Technology, Atsugi,

Kanagawa, Japan (e-mail: [email protected]).

Tetsuo Tanaka is with Department of Information and Computer Sciences,

Kanagawa Institute of Technology, Atsugi, Kanagawa, Japan (e-mail:

[email protected]).

Kazunori Matsumoto is with Department of Information and Computer

Sciences, Kanagawa Institute of Technology, Atsugi, Kanagawa, Japan (e-mail:

[email protected]).

identify hidden problems [1][2]. In the past, conventional

learning analytics mainly relied on off-line analysis, but more

recent developments have included the introduction of a

real-time learning analytics environment that can support

teachers and students in the classroom – made possible by the

spread of wireless LAN in the classroom and the adoption of

BYOD (Bring Your Own Device) where students bring their

own laptops to classes. Some research results have already been

reported in the literature [3][4][5].

The learning behavior logs used in conventional learning

analytics generally collect data on parameters such as the page

turning times associated with the learning material being used,

the amount of underlined content, and the number and type of

questions asked. As described in Ogata et al. [6], the learning

status of so-called active learners who actively underline, take

notes or ask questions can be usefully assessed using these types

of data. However, it is difficult to similarly assess the status of

many students who are attending classes but who may not be

actively learning.

Therefore, the purpose of this study was to examine whether

lectures could be improved through timely and appropriate

actions based on a better understanding of class attitudes and the

level of understanding of all students, including inattentive or

inactive students, in order to help them take in lectures more

efficiently.

In this study, we used the fill-in workbook system developed

by the authors [7] (hereinafter referred to as “this system”) in

order to collect logs of learning behavior from all students,

including those students who are not paying attention. Using

this system, all students in class are asked to fill in the blanks in

their workbooks, so that data on the learning activities of even

the most inattentive students can be collected.

In Section 2, we describe the topic of targeted lectures and

their associated problems, and then discuss possible approaches

to solving these problems in Section 3. Section 4 outlines the

lecture support system used when applying the fill-in textbook

approach. In Section 4, we also describe the results obtained by

analyzing the learning behavior logs collected using this system

in actual lectures.

II. TARGETED LECTURES AND ASSOCIATED PROBLEMS

In this study, we focused on the type of face-to-face lecture

classes that are commonly held in private Japanese universities.

Analysis of Learning Behavior Logs based on a

Fill-in Workbook System

- Toward Real-time Learning Analytics -

Kousuke Abe, Tetsuo Tanaka, Kazunori Matsumoto

T

INTERNATIONAL JOURNAL OF EDUCATION AND INFORMATION TECHNOLOGIES Volume 13, 2019

ISSN: 2074-1316 154

Classes are generally held in classrooms that accommodate

from 100 to 200 people and these classrooms often have tiered

seating as shown in Fig.1.

Especially in such large classes, students cannot always

maintain a high level of concentration over time. In order to

keep students focused, many faculty members use the “Monta

Method” [8] that leaves important words blank, as shown in

Fig.2, and displays them together with the relevant explanations

in the students’ workbooks. Having to fill in the blank content

displayed for each explanation shown in print can help students

continue to concentrate on the lesson.

Fig. 1 Lecture in classroom with tiered seating

Fig. 2 Lecture using the Monta method

However, in a class comprising more than 100 students,

especially in a classroom with tiered seating, it is difficult to

ascertain the status of all students - how many students are

actually filling in the blanks or working on exercises, for

example. Therefore, it is difficult to know when to take

appropriate remedial actions such as letting neighbors discuss

topics with each other or presenting them with problem-solving

exercises when the number of inattentive students increases.

In addition, it is difficult to assess a student's level of

understanding in real time during class. Exercises and quizzes

are effective in confirming whether students understand the

teacher's explanation, but these take time to score and count. As

a result, it is not always possible to take actions in a timely

manner, such as changing the details of an explanation

according to the degree of understanding exhibited by the

students. And if some students do not understand what is being

said, this may prompt them to whisper to each other and cause

other students to lose concentration and focus as well.

It is difficult for students to judge for themselves whether

they understand the content of a lecture, and how much they

understand or do not understand. In addition, some students

may think that they can understand the content, but they may not

actually understand it at all, or have an incorrect understanding

of it. This does not improve learning efficiency.

III. APPROACH

A. Extraction of Learning Attitude and Understanding Data

from the Learning Behavior Log

In this study, we used a fill-in workbook system developed by

the authors to collect learning behavior logs from all students,

including inattentive ones. In a class using this system, the

teacher makes the students fill in the blanks in a digital textbook

generated from a PDF document. As a result, students are forced

to interact with the system, and the learning behavior logs of

inactive students (students who do not actively take part in

learning actions such as underlining and writing notes) can be

acquired.

In addition, we analyzed the relationship between the

learning behavior log and the learning attitude / understanding

of students in the class, and assessed the learning attitude and

degree of understanding based on the data obtained from these

learning behavior logs. In classroom situations, the results can

then be visualized and presented to the teacher in a timely

manner. Here, the learning attitude is defined as the level of

student activity and involvement, and the degree of

understanding is defined as the degree to which the content and

meaning of the teacher's explanation is understood by the

student.

If teachers can effectively assess the student's learning

attitude in real time during class, they can then take actions that

encourage them to concentrate whenever the number of students

with a poor learning attitude is seen to increase. In addition, if

the degree of understanding can be assessed, it is then possible

to focus on improved explanations for topics that many students

may not understand. Alternatively, teachers can take other

actions such as giving hints during an exercise.

B. Learning Behavior Log

The overall structure of the system described above is shown

in Fig. 3. This system includes four modes: the authoring mode,

the browsing mode, the presentation mode, and the analytics

mode. Teachers can create teaching materials by uploading PDF

documents in the authoring mode and then adding blanks to

important sentences. Students fill in the blanks while referring

to these materials in the browsing mode. The operation is

recorded in the database as part of a learning behavior log. In

the presentation mode, the learning behavior log is presented to

the teacher in real time. The analytic mode is used for

retrospecting, and the teacher can refer to the learning behavior

log in three ways: “all students”, “individual students”, or

“pages”.

The items in the learning behavior logs collected using this


ISSN: 2074-1316 155

system are as follows:

- Fill-in blanks: Content and time of each answer written in

the blank, and whether answers are correct or incorrect

- Underlining: Underline position

- Notes: Note position and contents

- Operation logs: Page focus time, number of mouse

operations, number of keyboard operations per minute

- Page turning details: Time and destination page numbers

- Question details: Contents of questions asked by students

during class

In addition, a teacher's behavior log is also compiled and can

be used to compare course progress with the student's learning

behavior log. The teacher behavior log includes the following

items:

- Page turning details: Time and destination page numbers

- Deleted blanks (displaying correct answer): Deletion times

and target blank numbers

Fill-in Workbook System

Analytics

BEFORE CLASS DURING CLASS AFTER CLASS

• PDF doc• Blank data• Learning behavior

Upload docsAdd blanks

Student

PDF docBlank data

PDF docBlank data

Learningbehavior

Visualizedlearningbehavior

Teacher

Fill in blanksUnderlineTake notes

Teacher Teacher

Look backon the lesson

Assess the attitudeof students

Visualizedlearningbehavior

PresentationAuthoring

BrowsingStudent

Teacher

Fig. 3 System overview

IV. LEARNING BEHAVIOR LOG ANALYSIS

The analysis was based on data collected during 90 minutes’

of actual lectures conducted by one faculty member for about 90

students from the Department of Information and Computer

Sciences.

The 4 lectures used for the experiment were taken from a

15-part “Software Engineering” course on software

development methods. A faculty member created the fill-in

material in advance, using the authoring mode, and then gave

the lectures using the presentation mode. At the beginning of the

lecture, students were given a 5-minute explanation on how to

use the system. They then participated in the lecture using the

browsing mode. When using browsing mode, students filled in

the blanks on the screen, as shown in Fig. 4. (The blanks are

deleted once they have been filled in.) The students then

completed a questionnaire survey when the lecture was over.

In analyzing the results of this study, we focused on the degree

of synchronization between the teacher's slide transition, along

with the student's slide transition, the use of the note and mark

functions, how questions were asked, and off-screen time data –

as collected by the learning behavior log as an indicator of class

attitude.

We also examined the timing associated with the filling in of

blanks during the exercise (i.e., before or after the teacher

displayed the correct answer), whether an incorrect answer was

given before answering correctly, and the reference page being

used at the time of the exercise (i.e., whether or not the page

being referred to actually related to the exercise). These were all

used as an indicator of the student's level of understanding.

The following section describes how the class attitude and level

of understanding were extracted from the data collected in the

learning behavior log.

Blank

OpenedBlank

Fig. 4 Screen showing browsing mode in use

A. Extracting Learning Attitude from the Learning Behavior

Log

1) Reference Page Synchronization

In order to determine whether or not a specific student was

following the lecture being presented to the class, we analyzed

the synchronization between the page of the teaching material

being referenced by the student and the page being explained by

the teacher.

Fig. 5 shows the degree of synchronization recorded for each

student in the class. Each row represents the data recorded for

an individual student, along with the page that the teacher was

explaining and the actual time. The bar graph underneath shows

the number of students that were not in synchronization with the

teacher at any given time throughout the lecture. The students’

data shown in the figure are sorted by synchronization rate,

showing how closely they were synchronized with the teacher,

on a minute-by-minute basis. In the figure, any student cells that

are not synchronized for more than 30 seconds in any given

minute are shown filled in.

As shown in Fig. 5 (a), there was a clear division between

those students who were lagging behind the teacher, in terms of

page-turning timing, and those who were ahead. Also, as shown

in (b), there were many students who were not synchronized

with the teacher at all. Furthermore, as shown in (c), some of

these students remained unsynchronized for a relatively long

time.

Possible reasons for the delays observed could include

students taking notes on the previous page and being late

turning the page, or sleeping or doing other things and not

listening to the teacher's explanation. Possible reasons for going


ISSN: 2074-1316 156

forward could include moving on to the next page without

listening to the explanations given on pages that have few

blanks (or none at all) or simply proceeding onwards after

understanding the contents of the page being explained by the

teacher.

Therefore, synchronous data such as that described above

could be used to keep teachers informed, in real time, about the

number of students who have not understood explanations, or

have not caught up, or are moving forward ahead of the lecturer.

Teachers could then encourage students to concentrate more at

just the right time, or simplify their explanations, or adjust the

pace of their lessons accordingly.

However, these data, alone, do not provide enough

information to determine whether students are just lagging

behind or not listening, and in such cases it may be necessary to

consider other actions taken such as students “referring to the

previous slide” and perhaps taking note of the “keystrokes

made” by each student. These are issues that may need to be

considered in the future.

4 5 6 7 8 9 10 11 12 13 14 1516 17 18 192021 22 23 24 25

0 10 20 30 40 50 60 70 80

51

53

44

40

56

44

47

44

24

45

42

22

31

40

73

62

51

35

25

43

47

16

28

40

29

5051

45

43

41

48

4241

39

45

49

35

30

47

61

57

51

2221

15

1112

11

24

36

48

64

50

43

64

77

28

25

39

33

37

51

56

50

48

38

43

53

48

60

84

42

34

36

63

6768

43

25

23

34

44

0

10

20

30

40

50

60

70

80

15: 06 15: 16 15: 26 15: 36 15: 46 15: 56 16: 06 16: 16 16: 26

20

40

60

0

Page

Time

Students

Nu

mb

er o

f u

nsyn

chro

nize

d stu

de

nts

TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE

TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE

TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE

TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE

TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE

FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE

FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE

FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE

TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE

TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE FALSE

FALSE FALSE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE

TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE

FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE

FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE

TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE

FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE

TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE

TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE

TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE

FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE

FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE

TRUE FALSE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE

TRUE FALSE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE

FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE

TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE

FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE

FALSE FALSE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE

FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE

FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE

TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE

FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE

FALSE FALSE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE

FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE

FALSE FALSE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE

FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE

FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE

TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE




(c)

(a) (a) (a)

(c)

(b)

(c)

Fig. 5 Reference page synchronization

2) Unfocused Time

Another indicator that can help determine whether a student

is paying attention in class is the amount of time during which

the student is not focused on the system (i.e., when the system is

closed, when the system is open but the student is using other

applications, and so on).

Fig. 6 shows the amount of “unfocused time” recorded for

each student. The upper part shows individual student data and

the lower part shows a bar graph for the total number of students.

At the top, students' cells that are off the screen for more than 30

seconds in any given minute are shown filled in. (The horizontal

axis shows the time from the start of class.)

As shown in Fig. 6 (a), some students remained off the screen

for a relatively long time (3-4 pages of teaching material). These

students could have been doing other things, or even sleeping,

without listening to the teacher's explanation. Using this sort of

data in the classroom, alerts could be issued from the system to

encourage concentration.

As shown in Fig. 6 (b), there were also times when the number

of off-screen students gradually increased. This could represent

pages that have no blanks to fill in, or it might suggest that

teachers spent too long explaining the relevant course content.

In such cases, around 3 minutes after moving to the next page,

the number of off-screen students starts to increase. Based on

this data, if teachers are told that there are more students who

have not understood the explanations, teachers could take the

opportunity to encourage students to concentrate more.

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 26

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14

19 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 60 60 20 0 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 10 3 18 欠

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0

0 0 0 0 0 0 0 0 0 0 0 13 60 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 16 60 12 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 57

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 50 49

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 5 0 10 5 13 27 20 29 31 44 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 34 28 36 2 0 0 0 0 0 0 0 0 0 0 0 8 4 4 6 8 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45

0 9 60 9 29 15 0 0 0 0 0 46 59 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 19 0 0 0 0 0 0 0 0 0 0 0 1 0 24 2 17 4 0 0 0 0 0 16 50 4 6 1 59 11 0 0 0 0 0 0 0 0 1 0 0 29 0 4 0 0 6 25 0 0 1

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 46

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 20 54

0 0 0 0 45 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 12

37 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 50 9 4 10 2 60 6 39 0 0 0 欠

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 26 55 47 1 0 0 欠

34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35 26 4 0 0 0 0 0 0 0 20 59

欠欠欠欠欠 4 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 欠欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 59 59 60 60 2 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 3 0 0 0 0 欠欠 8 59 60 0 0 0 0 0 2 0 0 0 0 0 24 59 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 60 29 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 6 0 23 27

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 0 32 11 0 0 0 3 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 1 1 4 43 40 欠 32 欠

欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 11 17 0 0 欠

0 0 0 0 0 0 12 0 0 0 0 0 2 56 60 59 57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 13 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 27 60 59 0 0 0 0 18

欠 0 0 0 49 22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 欠 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 5 3 3 14 29 7 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 55 1 0 0 0 14 10 0 0 0 0 欠

欠 9 0 16 23 0 0 0 1 7 欠欠 0 0 0 0 0 0 0 0 10 0 9 50 0 28 56 60 15 0 0 0 0 0 0 0 0 0 1 0 0 0 1 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30

欠欠欠欠欠 5 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 9 12 21 1 0 0 0 34 23 57 31 0 0 0 0 0 0 0 0 6 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

11 8 0 0 0 0 14 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 13 7 0 27 11 0 0 0 0 0 0 0 0 1 54 0 16 32 11 0 0 0 0 0 0 0 6 60 18 0 0 0 0 0 45 17 47 0 0 0 29 14 0 0 0 0 11 25 0 0 6 59 60 59 18 23 48 0 欠欠

欠欠欠欠欠欠欠欠欠欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5

0 0 0 0 0 0 0 0 0 0 27 0 0 27 60 59 59 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 51 60 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23 59 34 欠 60 59 49 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22 27 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

欠欠欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0 46 59 60 19 0 0 0 0 0 0 0 0 0 0 0 0 20 59 60 60 59 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 58 23 16 53

0 0 0 0 0 0 0 0 0 0 0 0 0 59 60 60 1 0 0 0 0 0 0 24 0 39 60 60 40 60 60 16 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 58 60 18 47 0 0 0 44 58 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 欠 0 0 欠 0 欠欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠欠欠 0 0 0 0 0 0 33 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21 60 60 59 60 59 35 0 0

0 11 欠 0 31 10 0 40 0 0 0 0 25 0 0 18 40 0 0 58 51 0 0 0 0 0 38 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 60 59 60 47 18 60 60 60 60 59 28 15 34 27 29 12 0 0

22 59 60 60 60 60 28 25 0 0 0 0 0 6 60 25 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 10 0 0 0 44 51 17 11 40 59 44 60 60 54 3 0 0 0 24 20 41 24 0 0 0 0 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 10 0 0 0 0 26 58 47

44 60 60 31 59 30 35 60 21 0 0 0 0 54 60 60 5 0 0 0 0 0 0 0 0 0 11 59 37 59 42 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠

0 0 0 0 1 0 0 24 19 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 1 6 0 0 0 0 0 0 0 0 0 0 11 0 2 0 0 0 0 0 0 0 0 0 0 25 0 0 31 0 10 17 35 54 50 36 56 35 9 欠欠 50 36 1 35 60 59 59 60 60 59

10 10 60 34 59 19 1 60 60 2 9 27 26 60 60 54 43 60 13 0 0 20 0 1 19 2 53 0 26 0 45 43 13 0 0 0 17 12 0 1 0 0 20 0 0 0 0 2 0 57 59 50 19 43 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 30 45 0 0 0 0 0

12 47 59 35 60 31 55 59 26 16 59 12 0 42 60 59 53 0 45 19 38 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 27 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 22 34 0 30

欠 0 11 15 20 30 35 8 0 0 38 0 8 24 31 0 0 0 0 0 0 0 0 0 0 1 35 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 27 43 60 59 25 59 60 18 60 58 19 41 59 19 0 0 0 0 0 1 60 59 60 11 0 0

52 7 4 0 0 0 0 0 0 0 0 2 0 0 0 0 26 10 0 0 3 欠 2 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 2 1 5 0 21 17 4 0 0 13 50 0 2 0 0 0 28 60 44 31 9 22 10 34 4 45 6 14 31 23 56 33 20 16 14 46 60 49 2 欠欠

欠欠 60 8 60 52 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠欠欠欠欠欠欠 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 8 59 32 10 34 0 0

60 40 48 41 60 45 60 60 11 57 59 25 0 0 52 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 2 60 59 4 16 29 0 0 0 0 0 0 0 0 0 0 31 59 11 0 0 0 20 45 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 4 13 0 23 52

34 0 0 0 0 33 60 53 0 53 60 12 58 22 0 0 0 0 0 0 0 0 18 53 0 34 57 0 0 0 0 0 0 0 0 0 0 0 0 0 30 52 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 47 0 0 31 60 60 40 0 0 0 0 0 欠欠

51 0 51 40 59 44 48 48 0 0 0 0 0 0 0 4 0 0 0 1 60 12 8 50 0 0 0 0 0 0 0 0 0 31 48 27 0 0 0 0 0 0 0 0 50 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 57 60 46 60 59 13 0 0 0 0 0 0 0 0 0 6 0 0 0 0 42

0 0 14 5 45 14 60 60 32 60 53 57 3 0 0 0 59 60 36 0 50 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 28 11 60 38 2 6 55 58 0 0 51 60 40 60 32 0 0 0 0 0 0 0 0 0 0 0 2 60 37 21 6 14 16 4

41 38 26 33 28 47 24 40 39 39 38 40 31 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33 46 31 8 8 4 35 52 35 36 33 44 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 10 43 46 32 0 0 7 0 2

欠欠 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 60 60 59 60 60 60 59 60 59 60 60 60 59 欠欠欠欠欠

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17 0 26 0 29 54 0 0 0 0 0 0 0 0 43 60 60 59 60 0 0 53 60 60 8 0 0 0 0 0 0 50 60 60 47 60 44 0 0 0 0 0 0 0 0 0 0 0 0 31 27 20 6 0 0 49 59 60 14 36 54 0 3 54

18 40 18 41 58 3 44 40 18 59 39 6 49 55 58 59 56 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 12 0 0 0 11 1 0 0 0 15 32 21 0 4 53 0 20 60 60 57 56 44 1 0 欠欠欠

60 33 33 0 3 0 1 0 0 9 59 18 0 49 20 35 15 0 0 0 0 0 0 33 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 18 55 22 44 0 0 0 1 56 60 59 59 4 0 25 58 6 60 13 46 59 14 0 13 2 0 0 14 46 56 0 1 59 59 60 30 32 60 20 5 35

0 0 0 0 欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠 0 0 0 0 33 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 0 0 0 0 0 0 0 0 51 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠

59 60 59 59 59 60 60 59 59 60 59 41 59 60 60 48 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 60 59 60 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 60 4 0 0 0 7 60 59 60 59 60 59 59 41 0 0 0

欠 9 0 0 0 0 0 10 42 13 60 18 4 0 25 56 59 31 49 39 59 38 0 59 34 0 33 0 0 0 5 59 59 33 0 0 0 0 0 0 0 0 0 15 8 0 0 0 0 0 0 0 0 51 0 19 60 26 44 59 5 38 54 32 60 59 10 0 0 0 31 0 0 1 27 46 46 10 34 0 41 10

26 60 40 32 35 44 12 52 20 0 14 26 26 33 9 41 9 36 0 39 48 29 25 4 15 49 11 37 39 19 43 0 1 51 29 19 43 14 40 22 60 8 23 11 44 0 0 0 0 0 0 47 45 25 45 26 51 0 10 34 43 23 23 54 1 0 0 28 32 55 7 0 0 0 8 60 21 0 7 0 0 33

0 5 0 0 0 欠 29 15 40 52 24 13 15 42 15 44 23 37 57 56 30 0 28 54 58 60 50 38 54 60 59 31 0 0 1 60 60 59 59 46 30 32 38 55 0 0 0 0 0 0 22 21 31 7 51 0 0 0 14 34 45 32 44 10 44 12 31 29 25 0 15 0 0 0 0 0 0 14 0 0 0 2

37 59 60 59 58 8 50 0 0 31 60 60 59 59 59 60 59 60 38 21 59 60 11 0 48 7 59 60 60 59 60 59 59 60 59 60 59 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 49 2 17 0 0 0

欠 34 60 18 60 40 60 60 37 3 59 46 4 60 60 54 23 4 0 0 0 0 0 0 6 4 18 56 26 0 0 0 0 0 0 0 0 0 0 0 1 60 60 60 60 11 0 0 0 0 0 24 59 60 10 0 0 0 0 57 17 59 59 11 0 0 0 0 0 10 59 59 60 54 60 60 49 0 0 0 0 31

52 0 0 0 0 0 0 0 0 0 0 0 0 0 11 3 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

51 53 59 41 57 54 36 0 0 28 60 53 28 34 1 2 60 33 0 0 45 41 28 60 10 47 59 60 43 60 60 60 59 60 16 59 55 0 0 0 0 0 0 0 16 24 26 0 37 0 0 39 24 0 3 1 0 23 3 0 0 0 0 0 0 0 0 0 7 4 0 0 53 10 38 59 0 0 19 37 欠欠

48 16 38 60 48 55 60 52 21 60 54 20 60 60 60 60 60 51 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 0 0 8 58 60 52 60 60 60 33 36 59 40 60 46 60 60 37 59 38 51 0 0 0 0 0 0 0 0 0 0 37 18 60

0 0 0 0 0 0 0 4 10 20 0 0 0 16 59 59 60 38 0 51 59 10 25 60 13 27 60 59 59 59 31 0 0 50 39 0 0 0 0 22 30 59 54 34 41 0 0 0 0 0 0 0 0 18 28 53 33 0 0 18 59 43 36 59 60 28 55 60 43 60 59 60 60 59 59 60 59 59 59 7 0 0

39 60 41 49 60 欠 36 60 25 23 60 32 18 60 60 51 54 60 41 29 58 56 45 0 0 0 1 1 17 47 37 0 0 0 欠 0 0 0 24 24 31 48 26 0 5 0 0 0 0 0 44 47 20 52 47 33 58 0 21 25 0 0 13 28 60 26 0 0 0 40 37 0 1 60 31 0 0 22 0 0 19 欠

26 60 30 34 56 31 60 59 10 43 12 0 60 60 49 57 59 42 0 19 19 0 13 54 0 41 0 0 32 24 0 38 60 3 0 9 52 11 59 34 0 0 0 6 0 0 0 14 59 36 31 59 57 60 60 59 47 0 43 8 27 52 25 0 0 0 26 0 16 0 0 0 49 60 59 60 51 28 42 27 13 56

44 37 34 58 43 41 59 60 6 39 59 15 44 60 59 60 59 55 4 60 59 11 47 60 28 59 60 59 41 0 0 0 0 0 0 0 0 22 60 59 59 59 60 60 59 20 14 17 10 0 0 0 欠 0 0 8 0 0 0 0 43 60 6 0 0 0 0 31 30 0 30 59 60 59 32 8 60 27 52 5 0 0

14 欠 0 9 0 欠 2 0 0 43 60 14 43 60 54 59 60 57 2 0 0 0 0 0 0 0 57 49 50 59 60 59 52 欠 12 59 47 0 26 54 48 欠 16 0 0 42 23 15 0 7 8 7 8 5 29 0 1 30 欠 0 20 28 14 51 8 1 欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

59 60 59 19 59 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 19 17 59 60 40 45 59 60 59 60 59 6 0 0 0 0 7 59 59 60 59 1 0 0 0 0 0 0 48 60 59 60 59 29 34 60 29 60 56 43 60 59 31 59 60 41 55 59 60 59 60 60 48 37 59 16 0 欠

0 0 0 0 54 60 欠 60 0 51 59 35 50 59 59 56 58 58 48 0 0 0 0 0 0 47 60 59 60 43 0 0 39 59 55 57 60 29 44 60 59 60 60 60 60 59 22 0 0 0 39 59 30 0 49 59 60 25 0 48 0 0 0 0 0 0 0 38 41 0 24 59 59 60 59 60 10 0 0 0 0 25

欠 42 60 35 60 38 60 60 17 45 52 36 37 60 60 53 57 60 43 21 60 60 60 2 0 5 60 60 40 54 26 60 60 60 60 60 60 5 49 60 60 60 60 43 52 44 0 0 0 0 0 0 58 54 0 0 0 0 0 35 6 60 41 43 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

51 60 60 42 59 34 58 59 36 43 60 28 23 60 59 60 42 59 24 48 59 51 0 54 33 0 0 0 0 7 0 35 4 59 59 60 60 17 0 0 0 52 59 59 60 2 0 0 0 57 60 52 50 32 47 23 0 15 39 28 0 0 13 0 0 0 0 0 9 42 59 60 60 44 0 0 17 59 48 6 54 56

45 59 37 59 58 18 50 58 26 39 60 59 20 0 58 60 60 60 60 59 33 0 9 60 16 30 59 60 42 60 60 60 56 59 38 46 60 欠 0 0 0 0 40 60 60 1 0 0 0 0 28 59 59 59 60 60 60 58 0 21 59 60 60 3 10 56 44 0 0 0 0 0 0 0 29 23 4 3 35 0 0 欠

60 11 41 60 60 20 51 60 4 12 欠欠 58 欠 36 53 44 4 20 34 8 欠 18 0 0 39 2 53 60 56 60 26 0 3 53 60 59 15 26 60 60 60 60 60 42 0 0 1 0 3 0 欠 5 46 29 20 4 0 欠 36 19 57 43 57 60 47 20 60 44 25 23 4 56 49 47 37 1 0 30 45 欠欠

0 0 5 35 9 17 0 0 欠欠 3 2 0 0 0 5 0 0 0 0 0 1 1 欠欠 2 25 60 欠 10 0 0 4 0 0 8 0 0 欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

欠欠 29 47 欠欠 20 59 51 欠 21 29 44 2 34 60 60 59 49 55 60 50 欠 36 59 60 48 58 欠欠欠 8 23 32 60 59 60 40 3 0 5 43 59 59 51 欠 60 40 13 33 60 2 19 0 19 9 11 7 0 16 6 11 0 22 11 25 18 10 27 15 36 35 53 35 32 59 37 8 20 欠欠欠

欠欠欠欠欠欠欠欠 0 欠 0 0 0 46 60 59 60 41 0 欠欠 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠 0 0 0 28 60 59 59 60 60 59 36 0 0 53 60 欠 60 60 59 59 31 7 59 25 59 59 60 60 59 29 32 60 8 43 59 60 60 60 59 60 59 59 60 59 60

7 5 53 33 26 58 60 59 59 10 59 24 31 59 60 59 60 59 59 14 0 49 0 31 32 8 55 59 60 48 58 60 15 48 18 51 60 59 21 60 60 59 60 50 22 24 15 17 6 25 60 59 60 47 56 30 60 11 21 59 2 46 39 59 59 56 33 52 40 35 21 30 11 33 0 12 39 57 46 7 56 41

59 44 44 57 46 55 60 60 21 59 60 57 欠 53 60 58 59 60 27 56 60 36 16 59 23 欠 53 59 43 60 59 59 55 60 60 54 42 60 27 59 57 59 59 60 59 22 0 0 0 0 0 29 60 60 59 60 59 60 3 27 35 29 59 59 60 59 17 0 0 0 17 59 60 60 55 0 1 0 0 0 欠欠

欠欠欠欠欠 0 33 56 46 36 60 0 46 60 46 0 0 24 8 50 39 40 30 50 26 1 56 60 60 43 60 欠 39 60 60 60 60 欠欠 0 35 60 60 60 60 16 0 0 0 0 31 60 60 60 60 39 41 11 60 45 34 60 31 53 60 52 32 48 24 3 0 38 60 60 46 0 32 38 22 13 欠欠

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 欠欠欠欠欠欠欠欠欠欠欠欠 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

28 60 59 39 60 43 56 58 3 60 59 22 59 60 60 59 60 54 31 59 59 17 40 54 22 59 59 59 35 46 59 59 60 59 59 59 59 17 59 52 59 59 59 59 25 0 0 0 8 51 59 59 56 59 47 47 26 59 59 38 59 59 17 40 59 46 59 59 39 59 59 59 59 59 59 36 28 59 55 4 59 59

41 60 24 52 40 60 59 42 37 59 39 31 60 59 59 60 53 59 15 60 54 0 60 31 32 60 59 60 59 46 59 60 60 58 45 59 37 45 59 60 59 56 59 59 27 0 0 0 44 60 60 59 41 58 38 56 60 29 60 31 59 60 51 24 55 30 59 60 32 60 59 60 59 60 40 60 42 42 43 1 47 欠

欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠欠

59 60 60 60 59 59 60 60 60 59 60 59 60 60 59 59 60 60 59 59 60 59 60 60 59 59 60 60 59 59 60 59 60 59 60 60 59 60 60 60 60 60 59 60 60 59 60 59 59 60 59 59 60 59 60 60 59 60 60 59 60 59 60 59 60 59 60 60 59 59 59 47 60 60 60 59 60 59 60 60 59 59

4 5 6 7 8 9 10 11 12 13 14 1516 17 18 192021 22 23 24 25

0 10 20 30 40 50 60 70 80

Page

Time

Students

20

40

60

0

(a)

(b)

Nu

mb

er o

f o

ff-scree

n stu

de

nts

Fig. 6 Unfocused time

3) Summary of Learning Attitude Extraction

As described above, it was possible to determine whether

students were listening to the teacher's explanation closely and

were following the lecture, based on the synchronization

between the page being explained by the teacher and the page

being referenced by the student, and by the amount of off-screen

time recorded. In addition, we confirmed that when the

explanation of each page by the teacher was taking too long,

students did not listen to the teacher's explanation but referred

instead to other pages or left the system.

Therefore, by using this sort of data in a classroom situation,

it would be possible to notify the teacher accordingly or to alert

each student directly from the system.

In addition, as a means of after-class reflection, teachers could

use this data to identify places where students are not following

the lecture properly and where more students begin to do other

things. This information could then be used to improve future

classes. It could also be used to allow students to reflect on their

own attitude in class. How best to implement these support

functions will be a topic for further study in the future.

B. Extracting Degree of Understanding from the Learning

Behavior Log

This section describes the analysis carried out on the answer

times for the exercise page in order to determine the student's

level of understanding.

In class, all students had to complete 3 exercises, with the

teacher only displaying the correct answer after the students had

finished each exercise by themselves. In addition, a

multi-choice problem was described involving the selection of


ISSN: 2074-1316 157

the correct formula from a range of 4 possible choices.

Students who understood the lecture material could answer

correctly, without having to refer to the previous slide, before

the teacher displayed the correct answer. Students who had

some understanding of the lecture material could answer by

referring to the previous slide, and while students who did not

understand the lecture material at all could not answer the

question, they could still copy in the right answer once it was

displayed by the teacher.

In this system, when an answer was entered in the blank space,

the blank was deleted only when the answer was correct, and not

when it was invalid. Therefore, it was immediately known

whether the entered answer was correct or incorrect. In the

multiple-choice question described, even if the students didn't

understand the topic they could still answer correctly within 4

attempts by simply entering each possible answer in order.

Students who understood the topic could answer correctly with

a single entry, while students who possessed only partial

understanding usually needed to try several times before

answering correctly.

In order to analyze these results, we focused on the times taken

to fill in the blanks correctly during the exercise (both before

and after the teacher displayed the correct answer), whether

there was an incorrect answer given before answering correctly,

and the reference page in use at the time of the exercise (i.e.,

whether or not a page actually relating to the exercise was being

referred to) and used these parameters as an indicator of each

student's overall level of understanding.

1) Exercise answer times

In the experimental class, three test exercises were conducted.

The first was a descriptive exercise with 3 parts, the second and

third exercises involved answering multiple-choice questions

Fig.7 shows whether each student answered before the

teacher displayed the correct answer. The horizontal axis lists

each of the 3 exercises while each row shown above represents

the data for one student. The individual cells are filled in

wherever a student filled in the blank after the teacher had

displayed the correct answer.

Students who only filled in the blanks after the teacher had

displayed the correct answers had either not understood enough

of the course material to solve the problem by themselves, or

were not motivated enough to answer on their own. In such

cases, if it turns out that there are many students in class who do

not understand the course content properly, their teachers need

to explain the questions in more detail. If there are many

students who are not motivated, it may be necessary to change

the format to include questions that attract students' interest

more. This will help to improve classes.

In order to distinguish between a lack of understanding and a

lack of motivation, it may also be necessary to analyze whether

any additional keystrokes are made before displaying the final

correct answer.

2) Existence of incorrect exercise answers

Fig. 8 shows whether each student entered an incorrect

answer during the exercise. The horizontal axis lists each of the

3 exercises, while each row of data shown above represents the

data for one student. The individual cells are filled in wherever

the student entered an incorrect answer.

The presence of an incorrect answer suggests either that

students lacked understanding, or that they simply guessed

(incorrectly).

ex1-1 ex1-2 ex1-3 ex2 ex3-1 ex3-2 ex3-3

descriptive exercise multiple choice exercise

-141 -134 -75 -84 -17 -81 -387

-86 -80 -73 -114 -30 -105 -216

-138 -133 -103 -32 -44 -123 -433

-237 -226 -169 -270 -116 -165 -465

-99 -119 -62 -83 -37 -15 -250

-157 -138 -109 -66 -377 -598 -331

-151 -155 -106 -77 -392 -465 -519

-218 -213 -203 -407 -131 -57 -381

-148 -157 -133 -109 -61 -142 -463

-334 -332 -325 -402 -161 -207 -526

-1210 -1202 -2741 -3308 -2846 -2875 -3194

-131 -118 -92 -76 -39 -88 -354

-67 -57 -142 -19 -79 -67 -362

-200 -196 -166 -52 -92 -102 -421

-649 -641 -716 -1161 -445 -450 -764

-103 -115 -126 -90 -41 -74 -398

-195 -193 -173 -113 -51 -139 -457

-153 -136 -121 -126 -151 -135 -435

-342 -338 -334 -478 -847 -719 -991

-863 -852 -842 -1193 -1399 -1333 -244

-272 -269 -260 -74 -58 -126 -413

-98 -70 -45 -325 -482 -182 -507

-241 -228 -210 -89 -148 82 -237

-44 -38 -13 -80 6 -92 -402

-144 -136 -129 -415 -163 257 -57

-97 -89 -76 40 -57 -75 -307

-132 -138 -42 -135 -38 43 -270

-118 -108 -97 31 -297 -505 -812

-34 10 -22 -35 -104 -119 -417

-113 -105 -87 -129 -58 141 -248

-102 -87 -78 -61 -77 320 -9

-106 -80 26 -79 -162 -233 -488

-239 -233 -223 -263 -64 337 92

-79 -67 -45 -17 221 143 -167

-133 -111 -92 -66 -5 96 105

-152 -140 -126 -110 65 14 -308

-260 -254 -245 -302 -159 -0.999 -0.999

-151 -109 -68 -82 -8 361 105

-231 -228 -225 -364 -148 369 101

-127 -124 -88 -82 21 28 -288

-108 -97 -85 35 -81 196 -122

-204 -191 -146 -81 -28 221 105

-234 -227 -222 -41 -147 362 174

-274 -265 -257 -117 -430 456 183

-229 -226 -271 -241 137 161 -154

-113 -134 -41 7 35 99 -184

-208 -292 -238 -69 33631 33491 33143

-299 -289 -308 -400 254 347 61

-102 -71 -44 -93 56 147 58

47 50 56 -75 -65 -95 -302

-30 3 6 -89 -42 -98 168

17 20 42 -159 -617 -466 -869

-181 -177 -195 51 201 126 -190

-121 -67 -48 49 170 116 -28

-203 -198 -193 30 -112 234 84

-23 -28 36 -88 -142 247 104

-319 -317 -307 29 -139 351 121

-53 -15 19 -102 75 345 203

-38 -6 49 -66 105 332 60

-144 -111 -87 -0.999 -0.999 409 180

-92 -84 -70 7 8 342 53

28 30 35 15 -141 -17 -675

-304 -295 12 36 196 88 -233

-42 -29 -21 11 17893 17825 17608

13 18 26 -41 -30 99 -204

-73 -70 9 1 210 125 -189

-108 -102 -88 3862 2475 2381 2083

-76 -69 -61 22 18 324 66

48 52 56 154 -95 -189 -501

72 77 217 -76 123 107 -208

8 12 18 32 8 -87 -413

60 269 273 -54 676 580 315

31 33 -139 96 289 376 95

284 288 293 167 -142 831 -0.999

27 30 34 -43 671 376 257

45 47 52 -252 307 342 104

108 111 115 134 -12 357 59

100 101 108

Fig. 7 Exercise answer times

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 1 0

0 0 0 0 1 0 0

0 0 0 0 0 1 0

0 0 1 0 0 0 0

0 0 0 0 1 0 0

0 0 0 0 0 1 0

0 0 0 0 0 1 0

0 0 0 0 0 0 1

0 0 0 0 0 1 0

0 0 0 0 1 0 0

1 0 0 0 0 0 0

0 1 0 0 0 0 0

1 0 0 0 0 0 0

0 0 0 0 1 0 0

1 1 0 0 0 0 0

1 0 0 0 1 0 0

0 0 1 1 0 0 0

1 0 0 0 1 0 0

1 1 0 0 0 0 0

0 0 0 0 1 0 -999

1 1 0 0 0 0 0

0 0 0 1 1 0 0

0 1 0 0 0 1 0

0 0 0 0 1 1 0

0 1 1 0 0 0 0

1 0 0 0 1 0 0

0 0 0 0 0 1 1

0 0 0 0 0 1 1

1 0 1 0 1 0 0

1 0 1 0 0 0 1

0 1 1 0 1 0 0

0 0 0 0 1 1 1

1 0 1 1 0 0 0

1 1 1 0 0 0 0

0 1 0 1 0 1 0

1 1 0 1 0 0 0

0 0 0 1 1 1 0

0 0 0 0 1 1 1

0 0 0 1 1 1 0

1 0 0 1 1 0 0

1 0 0 0 1 1 0

1 0 1 0 0 1 0

1 1 0 0 0 1 0

1 0 0 1 0 1 0

0 0 1 0 0 1 1

0 1 1 0 1 0 0

0 1 0 1 1 1 0

1 0 0 0 1 1 1

0 1 1 0 1 1 0

1 0 1 -999 -999 0 0

1 1 1 0 0 0 1

0 1 1 1 0 1 0

1 0 1 0 1 1 0

1 1 0 0 0 1 1

0 0 1 1 1 1 0

0 1 0 0 1 1 1

1 0 1 0 1 0 1

1 0 1 0 0 1 1

1 1 0 0 1 1 0

1 1 1 0 1 1 0

1 1 1 1 1 0 0

0 1 1 0 1 1 1

1 0 0 1 1 1 1

1 1 1 1 0 1 0

1 1 1 1 1 0 0

1 1 1 1 1 0 0

1 0 1 1 1 1 1

0 1 1 1 1 1 1

1 0 1 1 1 1 1

0 1 1 1 1 1 1

0 1 1 1 1 -999 -999

-999 -999 -999 -999 -999 0 -999

1 1 1 1 1 1 1

01020304050

ex1-1 ex1-2 ex1-3 ex2 ex3-1 ex3-2 ex3-3

Fig. 8 Number of incorrect exercise answers

3) Reference pages used for the exercise

Fig. 9 shows whether other pages were being referenced

when completing the three exercises. Each row represents the

data for one student, and filled-in cells show when other pages

were being referred to.

Although there were many references to other pages in

Exercise 1, Exercises 2 and 3 could be answered correctly by

just entering all possible choices in order, without the need to

refer to any other pages (showing that a multi-choice exercise is

not really suitable for such experiments).

However, referencing other pages during Exercise 1, as

shown above, could be interpreted as signifying a lack of

understanding and that the student felt that the exercise could


ISSN: 2074-1316 158

not be completed without checking the previous page. It could

also signify willingness on the part of the student to solve the

problem without waiting for the teacher to display the correct

answer.

Those who entered the correct answer without referring to

other pages either answered correctly by themselves or simply

copied the teacher's correct answer. In order to distinguish

between the two possibilities, it was necessary to combine this

data with the answer times described in 4.1.1. If this could be

done in a classroom situation, teachers could give a useful hint

to students giving up without referring to other pages.

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

0 0 8.85

0 0 13.14

0 11.34 0

0 0 57.71

78.61 0 0

15.02 0 0

0 0 4

0 0 36.28

120.23 0 0

0 0 83.71

21.96 0 0

30.05 0 0

0 0 3.14

0 21.27 0

253.17 0 0

0 0 43.14

8.67 0 0

0 19.85 0

83.81 0 0

42.19 0 0

9.24 0 0

6.93 0 0

0 0 14

210.4 0 0

0 0 77.14

20.8 0 0

65.31 0 0

0 43.97 0

0 0 5.14

32.94 0 0

0 0 22.57

19.07 0 0

14.45 0 0

0 0 5.14

185.54 0 0

11.56 0 9.14

149.71 0 9.71

127.16 135.46 0

26.01 0 21.14

128.9 53.19 0

0 41.13 30.28

75.72 131.2 0

0 24.82 15.71

11.56 14.18 0

8.09 0 9.71

56.64 0 40.85

261.27 0 44.28

87.86 0 12.85

75.72 47.51 0

37.57 34.04 37.71

9.82 22.69 38.85

4.04 5.67 19.14

0102030

ex1 ex2 ex3

Fig. 9 Reference pages used for each exercise

V. CONCLUSION

In order to obtain a useful indication of the overall class

attitude and degree of understanding of all the students in a class,

even the inattentive or inactive ones, we analyzed their learning

behavior logs – using a fill-in workbook system developed by

the authors.

Specifically, we tried to discriminate between different

learning attitudes based on the use of reference page

syncronization and unfocused time data. Also, we tried to

determine the degree of student understanding based on answer

times, the existence of incorrect answers, and the reference

pages used for a series of exercises.

The results showed that it is possible to use this type of data to

identify whether or not students are paying attention to and

understanding class content at any given time. It was also

confirmed that the degree of understanding exhibited by

individual students could be usefully assessed using this

approach.

In future, we will examine the target data and the analysis results

further, and improve the method used for discriminating

between different learning attitudes and levels of understanding.

In addition, we will try to develop an effective presentation

method for feeding back the analysis results to teachers and

students in real time during class.

REFERENCES

[1] Olga Viberg, Mathias Hatakka, Olof Balter, Anna Mvroudi, The current

landscape of learning analytics in higher education, Computers in Human

Behavior 89, pp.98-110, 2018.

[2] Diana M. Naranjo, Jose R. Prieto, German Molto, Amanda Calatrava, A

Visual Dashboard to Track Learning Analytics for Educational Cloud

Computing,

[3] Atsushi Shimada and Shin’ichi Konomi, Hiroaki Ogata, Real-time

learning analytics system for improvement of on-site lectures, Interactive

Technology and Smart Education, Vol.15, No. 4, pp.314-331, 2018

[4] Atsushi Shimada and Shin’ichi Konomi, A Lecture Supporting System

Based on Real-time Learning Analytics, Proc. of CELDA2017,

pp.197-204, 2017.

[5] Xinyu Fu, Atsushi Shimada, Hiroaki Ogata, Yuta Taniguchi, Daiki

Suehiro, Real-time learning analytics for C programming language

courses, Proc. of the Seventh International Learning Analytics &

Knowledge Conference, pp.280-288, 2017.

[6] Hiroaki Ogata, Yuta Taniguchi, Daiki Suehiro, Atsushi Shimada, Misato

Oi, Fumiya Okubo, Masanori Yamada, Kentaro Kojima, M2B System : A

Digital Learning Platform for Traditional Classrooms in University, Proc.

Of The 7th International Conference on Learning Analytics & Knowledge

Understanding, pp.155-162 (2017).

[7] Kousuke Abe, Tetsuo Tanaka, Kazunori Matsumoto, Lecture Support

System using Digital Textbook for Filling in Blanks to Visualize Student

Learning Behavior, International Journal of Education and Learning

Systems, Vol.3, pp.138-144 (2018)

[8] Garr Reynolds, The "Monta Method" – Presentaion Zen,

https://www.presentationzen.com/presentationzen/2005/10/the_monta_

metho.html (2005)


ISSN: 2074-1316 159

Documents

Analysis of Learning Behavior Logs based on a Fill-in ...€¦ · The application of learning analytics is attracting increasing attention as one possible approach to helping solve