A platform for the Italian Bebras

Carlo Bellettini1, Fabrizio Carimati2, Violetta Lonati1, Riccardo Macoratti1, Dario Malchiodi1,Mattia Monga1, and Anna Morpurgo1

1Department of Computer Science, Universita degli Studi di Milano, Italy2Incanet s.r.l., Italy

{bellettini, lonati, malchiodi, monga, morpurgo}@di.unimi.it,riccardo.macoratti@studenti.unimi.it, carimati@incanet.it

Keywords: Contest Management Systems, Bebras, Informatics competitions

Abstract: The Bebras International Challenge on Informatics and Computational Thinking is a contest open to pupils ofall school levels (from primary up to upper secondary) based on tasks rooted on core informatics concepts, yetindependent of specific previous knowledge such as for instance that acquired during curricular activities. Thispaper describes the design choices, the architecture, and the main features of the web-based platform used tocarry out the Italian Bebras contest. This platform includes functionalities needed by students, teachers, andBebras staff during the execution of the challenge, tools to support the preparation of tasks and the trainingof students, instruments to evaluate the results and analyse data collected during the challenge. The platformis online since 2015 and it has managed the participation of around 25,000 teams and a significant amount oftraining sessions.

1 INTRODUCTION

The Bebras International Challenge on Informaticsand Computational Thinking (The Bebras Commu-nity, 2017) is a yearly contest organized in sev-eral countries since 2004 (Dagiene, 2010; Habermanet al., 2011), with almost two million participantsworldwide. The contest, open to pupils of all schoollevels (from primary up to upper secondary), is basedon tasks rooted on core informatics concepts, yet in-dependent of specific previous knowledge such as forinstance that acquired during curricular activities.

The tasks are designed by the Bebras community,which includes the representatives of more than 50countries. The community organizes an internationalworkshop yearly, which is devoted to proposing apool of tasks to be used by national organizers in or-der to set up the local contests. The national orga-nizers then translate and possibly adapt the tasks totheir specific educational context or to their specificway to propose tasks to the schools of their country.For instance, the French edition is based on interactiveversions of the tasks, and pupils can repeatedly sub-mit answers until they achieve a correct solution. Par-ticipation is individual in some countries and team-based in other ones; in some school systems the par-ticipation to the Bebras is compulsory. In any case

tasks should stimulate an entertaining learning expe-rience, thus they should be moderately challengingand solvable in a relatively short time: the commu-nity guidelines suggest three minutes on average. Un-fortunately, however, it is not easy to predict the dif-ficulty of tasks (van der Vegt, 2013; Bellettini et al.,2015; Lonati et al., 2017b). The difficulty of tasksis indeed a critical aspect of the challenge, as it is noteasy to predict, but crucial for the challenge’s success.Research on task difficulty in general, and on Bebrastasks in particular, is on going and data collection forstudying difficulties is one of our goals.

Besides being used during contests, Bebras tasksare more and more used as the starting points foreducational activities carried out by single teach-ers (Dagiene and Sentance, 2016; Lonati et al., 2017a;Lonati et al., 2017c). Bebras tasks were also used tomeasure improvements of students’ attitude to com-putational thinking (Straw et al., 2017).

In Italy (ALaDDIn, 2017) the Bebras Challenge isproposed to five categories of pupils, from primary(4th grade and up) to secondary schools, who par-ticipate in teams of at most four pupils. The set-ting described in this paper has been used since 2015;from 2009 to 2015 the Italian Bebras was managedby a different organization who operated through aMoodle-based system, now offline (Cartelli, 2009),

Figure 1: A typical task

and a similar contest (the “Italian Kangourou of In-formatics”, see below) was held in parallel: the twostarted to collaborate and eventually merged into thecurrent Bebras Challenge. The contest now takesplace in schools during the Bebras week (usuallyat mid November): under the supervision of teach-ers, teams access a web application that presents thetasks to be solved. Differently from some of theother countries, in Italy normally only few tasks arebased on multiple-choice questions; most of themare sophisticated tasks, in that they present open an-swer questions, they are interactive, or they requirecomplex, combined answers (Boyle and Hutchison,2009). Consistently, sometimes partial scores arecontemplated and there are no penalties for wrong an-swers.

An example of interactive task is presented in Fig-ure 1. The task has buttons which represent switchesthat selectively turn on or off rows or columns of askyscraper’s lights. The task’s request is to write asequence of switch operations to get the light pictureshown in the small skyscraper on the bottom left ofthe task, whatever the current state of the windows,which changes at every trial and is in fact unpre-dictable.

The 2017 edition saw the participation of 12,214teams (about 45,000 pupils, corresponding to around7 pupils every thousand who are attending an Italianschool, see Figure 2); the teams partook to the Bebrasduring the week November 13–17, each category had10 tasks to be solved within 45 minutes, for a totalcontest time of 50 hours across the week.

Before 2015, we organized a contest similar to Be-bras named “Italian Kangourou of Informatics” (Lis-soni et al., 2008). It had two rounds, the first onehaving the same format as the present Italian BebrasChallenge, which in fact at first inherited some of thetools already used for Kangourou. However, the in-

Figure 2: Bebras popularity in the Italian schools; the colorof each administrative region denotes the permillage ofpupils participating to the Bebras w.r.t. the total populationin school age (source of school data: National Institute forStatistics (ISTAT), 2014)

creasing number of participants and the evolution oftechnologies (the previous system was available as anapplication based on Macromedia Flash and workingonly on MS Windows) induced us to redesign and im-plement a brand new system which supports all thephases of the competition (before, during and after thecontest takes place) and that can be used on any plat-form with a recent web browser; this paper describesthe design choices, its architecture, and main featuresconceived to cope with Italian Bebras peculiarities.

The paper is organized as follows. In Section 2we present the general features of the platform and itsarchitecture. In Section 3 we discuss in particular thecomponents of the system that are in use during thecontest, and the underlying design choices. Section 4illustrates the life cycle of tasks which includes edit-ing, administration, evaluation, and analysis of col-lected tracking data. In Section 5 we survey tools usedto manage Bebras competitions in other countries andfinally in Section 6 we draw some conclusions.

2 PLATFORM DESIGN

The Italian Bebras platform supports all thephases of the competition.

Before the contest: organizers prepare the tasks;teachers sign up in the platform, register teamsand edit information about the school and theteam members (age, gender, optionally thenames); guests can try the tasks proposed in pre-vious editions of the contest and get immediatefeedback about their score;

During the Bebras week: teams access the tasksand submit their answers; organizers and teacherscan monitor the situation of teams;

After the contest: task answers are evaluated; col-lected data are analyzed by organizers; teams candisplay tasks together with the answers they sub-mitted, (one of) the correct solution(s), an expla-nation and some hints for further in-depth studyin a “It’s informatics” section, and the number ofpoints achieved for each task; teachers can displaythe total scores of all their teams and print atten-dance certificates containing the name of the teamand other optional information like the names ofteam members, the team’s scores, the global rank-ing.

Some aspects of the Italian situation we have todeal with must be pointed out. The software, hard-ware, and network ecosystem in Italian schools israther varied. Although we recommend to schoolsto use a recent version of one of two popular webbrowsers (Mozilla Firefox and Google Chrome), thedevices on which our Bebras platform is used are verydifferent with respect to form factors (several schoolsuse tablets), performances, and way of administering.For example, during the 2016 edition we saw connec-tions from several different operating systems: MSWindows 32bit (88.5%), GNU/Linux on Intel 32bit(2.8%), GNU/Linux on Intel 64bit (2.4%), MS Win-dows 64bit (2.2%), Linux on ARM (2.1%), and oth-ers (iPad, BlackBerry, . . . ). Hardware is often old andpoorly managed: it is common to have unsynchro-nized system clocks or misconfigured locales. More-over, very few schools have a reliable Internet connec-tion: we cannot assume the network is available dur-ing the whole contest or even only when the allocatedtime expires. All these issues have been considered inthe design of the Italian Bebras platform.

In the following we will use the following terms:a taskset is an ordered collection of tasks that areproposed to some category at some Bebras edition;whenever a team accesses the platform and down-loads a taskset, a test is instantiated, which gathersall information concerning the taskset (and hence theincluded tasks) and the team, together with answersand tracking data. Hence, for each team there shouldbe exactly one (running or concluded) test, and foreach taskset there are as many tests as the number ofparticipants for the corresponding category.

Figure 3 sketches the general architecture of theItalian Bebras platform.

The system manages different kinds of data: Tasksand Tasksets contains all information about categoriesand tasks, including texts, images, and code for ani-mation and interaction; Correction code defines how

tasks have to be graded; Answers & Tracking Data arecollected during the contest and include both the sub-mitted answers and other tracking data such as switchtime among tasks or editing of previously inserted an-swers; Teacher & Team Data contain all informationabout teams (e.g., members’ name, age and gender),schools and referring teachers.

Tasks&

Task Sets

Answers&

TrackingData

CorrectionCode

Teacher&

Team Data

Contest TeachersBackend

ContestHelp DeskCorrectionEditing

StudentTraining

DataAnalysis

Figure 3: The architecture of the server

In the upper part of the figure there are the accesspoints designed to be used by contestants: Contestis the main front end used by students to participateto the contest; Teacher backend is used by teachersto create, manage and monitor their teams; Studenttraining is used by students to look at and try tasksfrom the previous contests.

In the lower part there are the access points de-signed to be used by the Bebras staff: Editing is usedto collaboratively edit and code tasks, explanationsand solutions; Correction evaluates answers after sub-mission and assigns scores accordingly; Data Analy-sis performs statistical elaboration starting from col-lected tracking data; Contest Help Desk is used duringthe contest to monitor its evolution, detect problemsthat may arise, manage teachers’ help requests.

The darker nodes in the figure are the access pointsthat are active during the contest; they will be dis-cussed in the following section.

3 DURING THE CONTEST

3.1 Contest

The Italian Bebras is delivered via HTTPS: during theBebras week, each team connects to a website con-taining the HTML/CSS/Javascript for tasks (ten in the2017 edition) and tries to solve them within a 45 min-utes time limit.

To avoid a webpage look-and-feel and provide in-stead a videogame-like layout, tasks are prepared sothat each of them takes up exactly the space avail-able in the whole screen, independently of the device,browser, or screen resolution used. A little space issaved for the left navigation bar that allows teams toswitch among tasks, and the bottom navigation bar,which shows the remaining time and provides buttonsto open the help page and to exit submitting the an-swers. A screenshot of an interactive task is shown inFigure 1.

The contest scenario, described in the previoussection, introduces several important problems to besolved at a design level. The main problem is prob-ably the unreliable nature of the Internet connectionavailable in many schools. In particular this becomescritical when coupled to the strict time constraint forsolving the tasks. The first design choice is that allneeded resources (e.g., JSON with tasks’ definitions,images, and so on) need to be successfully down-loaded before the start of the test, so that the oper-ations of browsing and solving tasks become inde-pendent of the network. The browser’s Local Storagemechanism (W3C, 2016) is used in order to guaranteepersistence of data not only during the normal opera-tions of the test but also in case of computer’s crashes.When a team finishes its test or at the expiration of theavailable time, the system stops the interaction withthe tasks and tries to connect to the server in order tosend the answers. This operation has no timeout andin general can be retried also after a switch off-on ofthe computers, even in a following day. In some caseshowever this solution is not feasible, because admin-istration policies of the school labs may enforce a to-tal reset of all caches, cookies, and local storage ofthe browser at each switch on of the computers.

3.2 Help Desk

During the contest week, a phone helpline is availableto all the teachers, and Bebras staff members are con-nected to the help desk site to monitor the advance-ment status of tests. Due to the design restrictionsdescribed in the previous section, the visible statusinformation are limited to: start of a test (with all in-

formation regarding school, referring teacher, startingtime), end of a test (with information about used time,submitted answers and statistical data).

These simple pieces of information permit thestaff to individuate several anomalous situations:more than one access by a single team, out-of-timetests (e.g., time has expired but no answers have beensubmitted due to network problems or errors in theclosing procedure), prematurely closed tests (espe-cially for primary school level, some teams wronglyterminate the tests instead of passing to the next task).

In each of these situations the help desk staff com-piles a ticket, contacts the referring teacher, and ifneeded operates on the server status to reset anoma-lies. Teachers are guided in the operation to be ac-complished on the client (for example resubmittingthe answers if not correctly done the first time).

3.3 Teacher backend

Teams are enrolled to the Bebras challenge by theirteachers, who need first to sign up to the system. Thebackend allows them to create teams by specifyingtheir name and category and by editing the data aboutthe team members.

During and at the end of the contest, teachers canedit team data, for instance to rearrange teams in caseof absentees. They can also monitor their teams bylooking at a global table, and check whether their an-swers have been correctly submitted to the Bebrasserver. When something unexpected occurs (see thecases mentioned in the section concerning Help desk),they are notified via the backend and they can possi-bly contact the help desk staff.

After the end of the contest week, answers areevaluated and scores and placements appear withinthe team table in the teacher backend.

Attendance certificates can be then produced asPDF files and printed via the teacher backend. Eachcertificate includes the name of the team, the school,and the name of the supervising teacher; other datacan be optionally included team-wise, like the namesof team members, the gained score, or the overallplacement (given in terms of percentiles). Such op-tion is given in order to not stress the competitiveaspect of the Bebras challenge: indeed teachers usu-ally decide to make high placements public and avoidhighlighting low results. Certificates for supervisingteachers are also available.

More than a teacher from the same school mayenroll teams and, if they agree, the system allowsteachers from the same school to share informationabout their teams. Not requiring a unique referringteacher distributes the work load among several peo-

ple and permits to give credit to the many teachersin a school who engage their classes in the Bebraschallenge by training teams and/or supervising themduring the contest. Thus the enrollment of teams isfostered, which increases the overall number of par-ticipants.

4 LIFE CYCLE OF A TASK

We illustrate the functionalities of the platform bypresenting the development phases of the tasks thatare included in the Italian contest.

1. The tasks are conceived by a member of the Be-bras community, they are revised by other mem-bers, and they are finalized during the annual In-ternational Bebras Task Workshop.

2. A set of tasks is selected, adapted and translatedinto Italian; in particular some kind of animation,interaction or feedback is planned for most of thetasks.

3. Each selected task text is collaboratively editedand inserted into the Bebras platform; the edit-ing covers also the explanation and the “It’s in-formatics” texts, and the coding of the interactivecomponent of the tasks.

4. During the contest, teams access the platform todisplay the tasks and insert answers; when thetime expires, answers are submitted to the servertogether with tracking data.

5. After the contest, answers are automatically eval-uated and scores are assigned to teams; the logicsto evaluate answers and assign scores is usuallyimplemented before the contest.

6. After the contest week, students can look at theirscore, display tasks, and compare the answersthey submitted with the proposed correct solutionand related comments.

7. Answers and tracking data are analyzed in orderto study perceived difficulties, preferred tasks, dif-ferences among age levels, and other relevant is-sues.

Such phases are discussed in the following sec-tion, where the task in Figure 1 is used as leading ex-ample.

4.1 Task Editing

The platform provides a multilanguage (BBCode,Markdown, HTML) editor with live preview whichallows the Bebras staff to collaboratively edit and

code tasks, explanations and solutions. In particularit enables the staff to prepare and adjust the layout oftasks, create and edit texts, make tables, provide theJSON code for the interactive parts and the Javascriptcode to check the answers, this all directly within theplatform, so that each update on a task is immediatelyavailable to everybody.

Figure 4 depicts the Skyscraper task in editingmode: the layout of the task is shown and it is pos-sible to click on each part to modify it, with respectboth to its layout and its content. In this figure theinteractive part was clicked on and the pop-up menushows the object type and allows one to select whatto view and edit, in this case its JSON and the codeassociated with it. After selection of an option, a fullsize editor window pops up, which closes after sav-ing, uncovering the updated task.

By clicking outside the layout rectangles, a differ-ent menu pops up with options to access and edit the“checking program” and “explain and answer” (thatis the explanation and “It’s informatics”) parts of thetask.

Figure 5 shows the code windows for theskyscraper task. It is possible to modify it and to testit on a given answer.

Each Bebras task must have an explanation abouthow to solve it and an “It’s informatics” part whichillustrates the informatics concepts on which the taskis based. In the platform these two parts belong to-gether with the task. When accessing this part, theeditor opens a two-pane window, the top pane for thesource code, the bottom pane providing a live preview(see Figure 7).

Figure 4: A task in editing mode: the layout of the differentparts

Figure 5: A task in editing mode: the answer’s checkingcode

Figure 6: Tracking data about the accesses to the task andmodifications to the answer

4.2 Correction

Another important component of the system is thepart devoted to the correction of the tests. The de-sign solution takes into account several requirements:in order to guarantee the fairness of the contest nocorrection/solution hint must be available on the stu-dent client during the contest; on the contrary dur-ing training sessions the correction could be accom-plished client side; the answer verification and scoreattribution should potentially consider several factorslike partial solutions, category of the solver (the sametask could be offered to different categories but thescore is proportional to the relative difficulty), num-

Figure 7: A task in editing mode: the explanation and “It’sinformatics” parts

ber of trials or time used.The task editor makes possible (see Figure 5) to

write a Javascript function returning a pair of integerscorresponding to the actual and maximum score ofthe task. The system makes the context of the task(category, runtime and answer data) available to thefunction.

The Javascript language has been chosen as thepreferred language (the system permits to use alsoPython and PHP) because of the possibility to easilyexecute it on the client during offline training.

4.3 Data Analysis

In order to better understand and reason on contestresults, a significant amount of data is tracked duringthe contest.

Time of events (e.g., entering and exiting a task,clicking on buttons, changing text-area values) andvalues of changed fields are directly available as rawdata (see Figure 6).

These data permit us to calculate interesting met-rics: total time spent on a task, time dedicated tothe reading of a task before the first interaction, timespent before deciding to skip a task, successive revi-sion of an answer even after exploring other tasks.

Such data is processed to discover statisticalanomalies or other clues of cheating behaviours. Werecall that the contest is a non competitive challenge,but identifying these situations is needed in order toobtain clean and reliable statistical data.

Moreover, we use them to analyze the students be-haviour and better understand what attracts their at-tention and effort, by looking at how they peruse thetasks.

5 RELATED WORKS

Most of the countries involved in Bebras have theirown Contest Management System, often developedby the organizing group.

(Kristan et al., 2014) describes the Slovenian sys-tem, used also in Serbia. It is based on Yii (a PHP webframework, see (Yii Software LLC, 2017)) and it sup-ports multiple choice answers and interactive tasks,properly written in HTML/CSS/Javascript. They alsoprovide a task preparation system based on Django (aPython web framework, see (Django Software Foun-dation, 2017)) that can be used to create customizedcontests. The three-tier architecture of the Sloveniansystem is specially designed to be scalable and faulttolerant: it should be able to cope with a parallelparticipation of dozens of thousands of participants.They consider several levels of administrative roles inorder to manage all the life cycle of the contest: sys-tem, country, coordinators, and mentors. Persistenceof data is provided by MySQL databases (Oracle Cor-poration, 2017).

France has a highly modular system (France-IOI,2017) used both in Bebras and Algorea (an advancedcontest focused on algorithms). They have the largestuser base and their system is highly scalable on on-demand cloud computing platforms like Amazon webservices (Amazon.com, 2017). They also provide thereview system used by the Bebras community to setupthe annual workshop. The French system is imple-mented in PHP and tasks are to be manually written inHTML/CSS/Javascript, possibly using their own pre-defined libraries for common task types.

(Dagiene et al., 2017a; Dagiene et al., 2017b)describe the Lithuanian system. This system isalso based on a three-tier architecture implementedin MySQL, PHP and AngularJS (Google, 2017), aJavascript web application framework. Tasks are en-coded using HTML/CSS/Javascript and can be au-thored with the “Bebras lodge” editor, a separate sys-tem used also for achiving tasks. The contest man-agement system provides the ability to export tasks inthe SCORM format (Advanced Distributed Learning,2017), a popular standard used by many e-learningplatforms.

6 CONCLUSIONS

This paper presented the web-based platform support-ing the Italian edition of the Bebras challenge, whichis used by participants and organizers before, duringand after the contest. The platform core has beenonline for the last three years, managing the partici-

pation of around 25,000 teams, as well as a signifi-cant amount of training sessions. The platform is akey asset for the Italian Bebras, it enables the man-agement of a contest involving thousands of studentswith a very lean operational staff: five academics andone professional programmer who collaborate in theirspare time, and 500 hours of helpdesk support mainlydelegated to junior collaborators.

Several features have been added during its threeyears of activity, which can be summarized as fol-lows: tasks access is web based, but the system isquite robust since it keeps working even with pooror intermittent Internet connections, which are quitecommon in Italian schools, and allows one to re-cover data in case of client crashes; tasks can be dis-played from a browser running on a variety of dif-ferent devices, spanning from computers with low-resolution old screens to last-generation tablets andsmartphones; when rendered each task occupies ex-actly the size of the screen, independently of the useddevice, so that the contestants’ experience is moresimilar to videogaming than web browsing; besidesdata about team members and submitted answers,during the contest several other data are tracked thatconcern the interactions of users with the platform,thus enabling a thorough analysis of results, also inlight of a better assessment of the difficulty of eachtask (Bellettini et al., 2015; Lonati et al., 2017b);management of teams by teachers is integrated withinthe platform; help desk tools are available to moni-tor the contest and support teachers whenever prob-lems arise; finally, all the information related to agiven task (i.e., text, images, and code for anima-tion/interaction/feedback and for automatic evalua-tion) is embedded into a JSON encoding, thus allow-ing the possibility of exporting tasks.

We plan to extend further the analyses of track-ing data, in order to better understand how studentsengage with tasks. Moreover, we want to exploitthe modularity of our platform to implement anothercomponent of the Bebras platform, that will enableteachers and students to make use of the tasks beyondthe contest, for instance by accessing tasks individu-ally out of specific tasksets, exporting them in print-able forms, or creating personal tasksets. We hopethis will foster the use of tasks in curricular activi-ties, as educational resources to learn informatics andcomputational thinking.

ACKNOWLEDGEMENTS

The authors thank the international Bebras commu-nity and all the schools which took part in the contest.

REFERENCES

Advanced Distributed Learning (2017). Sharable con-tent object reference model. http://adlnet.gov/adl-research/scorm/. Last accessed on November2017.

ALaDDIn (2017). Bebras dell’informatica. https://bebras.it. Last accessed on November 2017.

Amazon.com (2017). Amazon web services. https://aws.amazon.com/. Last accessed on November2017.

Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Mor-purgo, A., and Torelli, M. (2015). How challengingare Bebras tasks? an IRT analysis based on the per-formance of Italian students. In Proceedings of the20th Annual Conference on Innovation and Technol-ogy in Computer Science Education ITiCSE’15, pages27–32. ACM.

Boyle, A. and Hutchison, D. (2009). Sophisticated tasks ine-assessment: what are they and what are their bene-fits? Assessment & Evaluation in Higher Education,34(3):305–319.

Cartelli, A. (2009). Il sito italiano della compe-tizione internazionale ”il castoro”. http://www.competenzedigitali.it/castoro-i.htm. Cur-rently offline.

Dagiene, V. (2010). Sustaining informatics education bycontests. In Proceedings of ISSEP 2010, volume 5941of Lecture Notes in Computer Science, pages 1–12,Zurich, Switzerland. Springer.

Dagiene, V. and Sentance, S. (2016). It’s computationalthinking! bebras tasks in the curriculum. In Proceed-ings of ISSEP 2016, volume 9973 of Lecture Notes inComputer Science, pages 28–39, Cham. Springer.

Dagiene, V., Stupuriene, G., and Vinikiene, L. (2017a). In-formatics based tasks development in the bebras con-test management system. In Damasevicius, R. andMikasyte, V., editors, Information and Software Tech-nologies: 23rd International Conference, ICIST 2017,Druskininkai, Lithuania, October 12–14, 2017, Pro-ceedings, pages 466–477, Cham. Springer Interna-tional Publishing.

Dagiene, V., Stupuriene, G., Vinikiene, L., and Zakauskas,R. (2017b). Introduction to bebras challenge manage-ment: Overview and analyses of developed systems.In Dagiene, V. and Hellas, A., editors, Informaticsin Schools: Focus on Learning Programming: 10thInternational Conference on Informatics in Schools:Situation, Evolution, and Perspectives, ISSEP 2017,Helsinki, Finland, November 13-15, 2017, Proceed-ings, pages 232–243, Cham. Springer InternationalPublishing.

Django Software Foundation (2017). Django. https://www.djangoproject.com/. Last accessed onNovember 2017.

France-IOI (2017). France-ioi software ecosys-tem. https://france-ioi.github.io/projects-overview/readme/. Last accessedon November 2017.

Google (2017). AngularJS. https://angularjs.org/.Last accessed on November 2017.

Haberman, B., Cohen, A., and Dagiene, V. (2011). Thebeaver contest: Attracting youngsters to study com-puting. In Proceedings of ITiCSE 2011, pages 378–378, Darmstadt, Germany. ACM.

Kristan, N., Gostisa, D., Fele-Zorz, G., and Brodnik, A.(2014). A high-availability bebras competition sys-tem. In International Conference on Informaticsin Schools: Situation, Evolution, and Perspectives,pages 78–87. Springer.

Lissoni, A., Lonati, V., Monga, M., Morpurgo, A., andTorelli, M. (2008). Working for a leap in the generalperception of computing. In Cortesi, A. and Luccio,F., editors, Proceedings of Informatics Education Eu-rope III, pages 134–139. ACM – IFIP.

Lonati, V., Malchiodi, D., Monga, M., and Morpurgo, A.(2017a). Bebras as a teaching resource: classifyingthe tasks corpus using computational thinking skills.In Proceedings of the 22nd Annual Conference on In-novation and Technology in Computer Science Educa-tion (ITiCSE 2017), page 366.

Lonati, V., Malchiodi, D., Monga, M., and Morpurgo, A.(2017b). How presentation affects the difficulty ofcomputational thinking tasks: an IRT analysis. In Pro-ceedings of the 17th Koli Calling International Con-ference on Computing Education Research, pages 60–69. ACM New York, NY, USA.

Lonati, V., Monga, M., Morpurgo, A., Malchiodi, D., andCalcagni, A. (2017c). Promoting computational think-ing skills: would you use this Bebras task? In Pro-ceedings of the International Conference on Infor-matics in Schools: Situation, Evolution and Perspec-tives (ISSEP2017), Lecture Notes in Computer Sci-ence. Springer-Verlag. To appear.

Oracle Corporation (2017). MySQL. https://www.mysql.com/. Last accessed on November 2017.

Straw, S., Bamford, S., and Styles, B. (2017). Randomisedcontrolled trial and process evaluation of code clubs.Technical Report CODE01, National Foundation forEducational Research. Available at: https://www.nfer.ac.uk/publications/CODE01.

The Bebras Community (2017). The Bebras internationalchallenge on informatics and computational thinking.https://bebras.org. Last accessed on November2017.

van der Vegt, W. (2013). Predicting the difficulty level of aBebras task. Olympiads in Informatics, 7:132–139.

W3C (2016). Web Storage, second edition. W3CRecommendation http://www.w3.org/TR/2016/REC-webstorage-20160419/.

Yii Software LLC (2017). Yii framework. http://www.yiiframework.com/. Last accessed on November2017.