Computer Supported Authoring of Questionnaires

Willie D. Leiva1 , Christian M. Adriano1, Paulo C. Masiero

1 Supported in part by FAPESP

F.A. Willie Dresler Leiva, ICMC USP, wdl@icmc.sc.usp.br

����������� � �������������������� �����!��"����#���%$&"('�)*'���+*"�,�-�"��������.��/�+102+�)

�*"(��+ �.+ �13�"4�5��+�6%��7&,�"�+*��/8������/#�9�!,:�!� "(�;�����.��/#+��1<!,�"(���=��+ +�"(+ +���"����?>�@7*�*"(�A��"�B8��"$&�.�C/#+����D"���9����<C�*"��4�*����-?+&7*�8�D��+ +1"�+ +��D"�C�E�9<��*3��8������/�F$@"4��'!+�"��GB8"�,D�*3���� �����H0#��7 �13�"(�I��� "J���9� 7K-�"L��.'�-�"A"(���9<�/�3M,�<C"@�*3�"7*�9-.-.��$�����/%��+*�C"��G�N+GOQP ��+�"J��7=�����*"(/8�(���?�.�9�R�*�#��-�"�����������/S+102+��*"(�T���;,�*3C"%����+ +���'���-.���50U�*�M�����C-�"V�D"(���J��<��*3C�8��)N,�"(7?�.��"�,W�?08��"�+��97�X�<�"(+��?�.���C)�;���?� "�+*>CYZ"4����+�"�,:�*3�"A3H08���9�*3�"�+��.+[�*3!���E�%X�<C"�+1�*�������;�(�K��")���<��*3C�8���.��/�*�9��-.F\7*�8+1�*"�� "�,]'�0M���]^@_U`a,�"�7*�.��"�,S-K�9��/�<!�9/�"(Fb$@��<C-K,S�!���(B��K,�"D�*3�",H"�+��.��"�,]7*-."�L8��'���-.���?0%$@3C��-."�3C"-����.��/]�*�%�.�HB8"�+��?�./��(�*"%�M��"(�8+����;��'�-�"�/8"()�C"����.�4�D�8,�"�-b7*�8�JXH<�"�+��?�������;���.��"V+�>C�A+&�J�;� "(-��.�D�.�!�9� 0%��"V+�<�-.�?Fc$E"J����)�C-."(��"(���*"�,��9�#�9<��*3C�8���.��/4�*����-b�13!�(�c$A�8+d�����*"�/2�(�(�*"�,D�*�J�4$@"'C)*'���+�"�,-�"��9��C�.��/e+102+��*"(�JF@���9��"�,gfd�@` _R>&hE"(���*3C"V�]�*"�����3�"��%�C�8�S+��K<;,�"(���N+�C"V"�,4�*��6����($Z��,�B2�9�!��"�,J�������C<��*"(�=�����!��"(���*+c�*�@<;+*"[�*3�"[�*����-�>i�j!k�lNman�l�o�pAq

r[s�tDu�vCw*x�y z*{U|9}!|9~�x��e� }!�1wNyGv!��w?�.s�}C�=�@vCw*��s8yG�.}C~s��\�4v�x��1w*��s�}�};|(�Ky�xV���

I. INTRODUCTION

The design and authoring of assessment is an issue faced inmany web-based learning systems. Several approaches areavailable, but most consist of solutions tightly coupled to therespective learning systems. Such limitation is a serious diffi-culty while implementing and testing different assessmentapproaches, as for instance, some qualitative assessmentbased on augmented teacher-student interaction. A comple-mentary issue is the lack of a standard for modeling ques-tionnaires.

In this paper, we pose the hypothesis that a questionnaire-authoring component, fostered by an XML (eXtendedMarkup Language) [1] defined language, would provide thedesired flexibility while helping to investigate a reasonablegeneric model for questionnaires.

The strategy comprised the design and implementation of acomputer tool for the authoring of questionnaires, the Quest-CALM — Quest from Questionnaire in the CALM system(Computer-Aided Learning Material [2]). The CALM systemis a platform of software components to deploy web-basedlearning systems [3]. The current implementation of CALMcomprises an intelligent tutoring system [3] to recommendlessons and some interactive tools, such as a web page anno-tation mechanism (group and personal), a software testingcomponent, an applet configuration interface, and a chat. Thepedagogical orientation adopted was inspired by the GoalBased Scenario approach [4].

The next section outlines the major features found on toolsfor questionnaires authoring. Section III describes advantagesof QuestCALM over these tools. Section IV shows thegraphical interface of QuestCALM. Section V details somebenefits of integrating QuestCALM with CALM. Finally,Section VI presents some conclusions.

II. TOOLS FOR QUESTIONNAIRES AUTHORING

The �C� ����� tool [5] creates and automatically grades mul-tiple-choice and true-or-false self-testing quizzes. The re-sulting document may be available over the WWW (WorldWide Web). Authors do not need to know HTML to use thistool.�]���N�=�����������

[6] is a suite of five authoring tools for cre-ating multiple-choice, gap-fill, text-entry, jumbled sentence,and crossword exercises. These exercises are fully interactiveand compatible with all languages based on the roman char-acter set (including accented characters). As in qform, Hot-Potatoes tools are designed for non-programmers, and theproduced exercises may be available over the WWW.�:�b�2���N�����?�*�;�8 

[7] allows the authoring of the followingtypes of questions: Open, matching, multiple-choice, andtrue-or-false. The correct answer is included in test-basedquestions; therefore, answers can be automatically evaluatedwhen a student finishes a questionnaire. If open questions areused, the answer is mailed to the instructor whose electronicaddress was included in the question.¡:¢b£�¤2¥&¦�§�¨

[8] allows a student to answer online a multiple-choice or true-or-false quiz. At the end of the quiz, it is pos-sible to present the following information: Answers for thequestions answered incorrectly, the percentage score, previ-ous high scores on the quiz, and a grading scale. QuizTestalso emails detailed results to the instructor, along with thedate and time of the solved quiz.©«ª�¬®­4¯�°®±8²Gª

[9] supports the following types of questions:True-or-false, multiple-choice, free answer, and gap-fill.³«´�µ®¶J·

[10] is one of the most complete tools for devel-oping courses over the WWW, providing the instructor withmany statistic data such as course offerings and student indi-vidual progress. WebCT uses multiple-choice, true-or-false,and open questions.

III. FEATURES OF QUESTCALM

QuestCALM enables not only the authoring of multiple-choice, open, and true-or-false questions, but also questionsbased on external tools. For example, this last type of ques-tion may facilitate the assessment in a music course, provid-ing that a special-purpose application be dispatched to show amusical notation. The diversity of question formats supportedby QuestCALM was stipulated as a first attempt towardsqualitative assessment within a web-based learning system.

Unlike previously reviewed tools, QuestCALM is easilypresented either in English or in Portuguese languages. Otherlanguages may also be added without changing the source

code. As in all reviewed tools, it is unnecessary to know ad-vanced computer concepts in order to create a questionnaire.

All question categories within QuestCALM share some at-tributes such as statement, weight, and a hint for resolution.The hint may be either a text or a link to a WWW document.

Questions were implemented with an XML-based lan-guage. XML is a valuable resource not explored by the othertools. It allows incorporation of other types of questions. Newattributes may be incorporated to already existing questionsor to a specified set of questions.

IV. A PRACTICAL EXAMPLE

When QuestCALM is activated, the window showed inFig. 1 is displayed. The ¸:¹bº2»�¼N½?¾�¿b¿®À�½*Á9º menu has the fol-lowing options: Â4Ã�Ä , Å�Æ;Ç8È , ÉbÊ�Ë�Ì , and ÍbÎ�Ï�ÐUÎCÑ�Ò.Ò�Ò The ÓAÔ;Õ�Ömenu enables to change and remove an existing question,while the four buttons at the menu bottom are used to create aquestion.

Fig. 1. The main window of QuestCALM

Questions may be configured using the QuestCALM inter-face. Fig. 2 shows how to specify attributes of an open ques-tion in QuestCALM.

Fig. 2. Specification of an open question

Fig. 3 exhibits the configuration window used to create atrue-or-false question.

Fig. 3. Creation of a true-or-false question

Fig. 4 illustrates how a teacher may incorporate a table intothe student’s answer by specifying the full filename of aneditor.

Fig. 4. Question based on an external editor

Fig. 5 demonstrates the first window of a multiple-choicequestion.

Fig. 5. The main information for a multiple choice question

Selecting the “New choice” button in Fig. 5, the windowrelated to Fig. 6 is displayed.

Fig. 6. Edition of one option of a multiple choice question

Questions constructed so far are presented in the mainwindow of QuestCALM (Fig. 7).

Fig. 7. The finished questionnaire

V. INTEGRATION OF QUESTCALM WITH CALM

The integration of new tools to the CALM platform bene-fits a lot from the component-based architecture adopted.Design and development of component-based educationaltechnology is a current thread of research [11] and discussion[12]. The IEEE Learning Standards Technology Committee[3] defines the concept of "learning object", which, for ourdesign purposes fits well the term component. Here followsthe definition for learning object: "Any entity, digital or non-digital, which can be used, re-used or referenced during tech-nology supported learning".

Since the CALM platform required and enabled Quest-CALM to be developed as an independent software compo-nent, a very refined function and interface interoperabilitywas accomplished. Visually, a specific button in the CALMgraphic interface gives the access to the QuestCALM tool.Logically, the QuestCALM tool exchanges events and datawith other components of the platform, while obeying to apre-defined framework of integration.

We advocate that component-based construction of web-based learning system promotes the exchange of reusablesolutions among developers and researchers. From our expe-rience, other advantages relate to better design, faster devel-opment, and ease of integration.

VI. CONCLUSIONS

QuestCALM was our first step in investigating an impor-tant facet in web-based learning systems, the assessment de-sign. The tool is appropriate, exactly due to its fully integra-tion to an web-based learning system. Another point is itsflexibility of not restricting assessment to a fixed set of for-mats, since other types of questions may be created. Further-more, the language option for its menus and windows is eas-ily configurable by a text file for each new language.

Our next efforts are to develop a complete learning sce-nario for qualitative assessment. The general approach will beto support the indispensable interaction between teacher and

student by means of web page annotation mechanism, whichis already available with the CALM platform. Student andteacher communicate understandings, corrections and argu-ments via annotations at conflicting points of an answer.Such process lasts until they feel satisfied with the final an-swer. In this way, we expect to give a further step towardsqualitative assessment in web-based learning system.

REFERENCES

[1] B. Bos. ×CØ�ÙÛÚ Ü=Ý(Þ Ücß!à(Ú ÜVá â*ã 2000. http://www.w3.org/XML/1999/XML-in-10-points.

[2] C. M. Adriano ä å(æ(ç . “ Inquiring the Course Paradigm with CALM” .è!é*ê(ë ì�êîíQï ð(ñ�ò1óVï ñ é*ó(ô(ï õ ê(ó(ôVö(÷;ê(ó.í ñ é*ñ ó(ë ñbê�ó;øbó ù8õ ó(ñ ñ é*õ óNù&ô(ó(ú ÷!ê(û�ü�ý(ï ñ éøbúVý(ë ô(ï õ ê(ó(ì Rio de Janeiro: 1999.

[3] Learning Technology Standards Committee. þ®ÿ�� ��� ���� �� ������� ÿ������� �������� ������ ���� "! #�$�%�%�% . http://l tsc.ieee.org/wg3[4] R. C. Schank, A. Fano, B. L. Bell, M. Y. Jonea, “The Design of Goal-

Based Scenarios” , &�'�(�)+*�,�-�.�/�0�*2143 '�(�56(�/�-�.�7 .�8:9<;�7 (�.�;�(�= , vol. 3(4), pp.305-345, 1993-1994.

[5] B. Cunningham. ><?�@4A B�C�D�EGFIH�JLK�H�MN@�O 1995.http://www.satlab.hawaii.edu/ space/hawaii/qform.html

[6] H. Martin, and A. Stewart. “HotPotatoes: Free Tools for Creating Inter-active Language Exercises for the World Wide Web” , in PRQ�S�T�U�V�W WYX Z�[(“From Classroom Teaching to World-Wide Learning” ),www.arts.kuleuven.ac.be/eurocall98/abstracts/holmesmartin.htm Bel-gium, 1998.

[7] M. G. C. Pimentel, J. B. Santos Jr., and R. P. M. Fortes. “Tools forAuthoring and Presenting Structured Teaching Material in the WWW”,in \�]�^L_�]�`ba c�d . 1998. http://www.icmc.sc.usp.br/~mgp/webnet98.

[8] K. L. Pfaff-Harris. e+f�g�h iIj k�l�m�gonRp�q�f�r�j m�g�ktsouvq�h w�x�y�k�j2z 1999.http://www.linguistic-funland.com/scripts/QuizTest/

[9] R. H. Scapin. {�|�}�|�~ ���I�v|������2�4��������~2�����6����|���� � ������������������|���� � �����2��6� ����6� ��� ��|:�v����~ ���L��� ��|��L�v|�� . São Carlos: Institute of Physics of SãoCarlos at USP, 1997. 111 p. (MSc Dissertation in Portuguese).

[10] M. Goldberg, S. Salari, and P. Swoboda. “World Wide Web CourseTool: an Environment for Building WWW-Based Courses” , in �R� ��� �R������ ������ �� � ¡��� �¢¤£¡���¢ ¥¦£�� ¥��:£v��§�¨�¡��2����������©�� . Paris: 1996.

[11] C. Quinn. Learning Objects and Instruction Components. ª�«�¬ ­�®�«�¯�¬ ° ±�«�¯�²³ ±�®�´�µ¶±2·I¸R¹�´�º�¯�¬ ° ±�«�¯�²+»�­�º�¼�«�±�² ±t½�¾¦¿ÁÀ+±�º�° ­�¬ ¾ ,http://i fets.ieee.org/discussions/discuss_feb2000/, discussion held 14-25February, 2000.

[12] K. R. Koedinger, D. D. Suthers, K. D. Forbus, Component-based Con-struction of a Science Learning SpaceÂLÃ6Ä�Å�Æ Ç�È�Ä�É�Ê�Æ Ä�ËIÌ Í�Î�Ê�ÏÑÐIÇ�Æ Ä�È�Ò+Å�Ì ÓÄ�Í�Å�Ä vol. 1452, pp. 166-175Ô�Õ�Ö�×�Ø�Ù ÚRÛ�Ü�Ù Ý�Þ Þ ß àNÝ�Ü�Ù+á�×�Ù Ö�Ø�ß Ü�à:â�ãNä�Ù Ý�åä6æ�Ö�Ü�çè Ý�Ø�Ý�Ü�é�Ý , San Antonio Texas, 1998.