Instruments For Assessing Learning Achievement Some …unesdoc.unesco.org/images/0010/001016/101628E.pdf · Instruments For Assessing Learning Achievement Some Basic Considerations

Studies and Working Documents

No.4

Instruments For Assessing Learning Achievement

Some Basic Considerations

by

Susanne SchnüttgenUNESCO consultant

Division of Basic EducationFebruary 1993

--------------------------------------------------------------

The views and opinions expressed in this working document are those of the author anddo not necessarily reflect those of UNESCO

Optical Character Recognition (OCR) document. WARNING! Spelling errors might subsist. In order to accessto the original document in image form, click on "Original" button on 1st page.

The document Instruments for Assessing Learning Achievement: SomeBasic Considerations (no. 4) has been commissioned by UNESCO in theseries of Studies and Working Documents for the Joint UNESCO-UNICEFInternational project Education for All Goals: MonitoringLearning Achievement.

This document is prepared by UNESCO consultant Susanne Schnüttgen, whohas been associated with this Project since its initial phase. She has carefullyfollowed and monitored the progress and consultations surrounding theconceptualization and direct application of the major instruments of thisInternational Project (tests and questionnaires).

In this document, Ms. Schnüttgen highlights some of the basic conceptual,methodological and practical issues confronting test construction andquestionnaire development. Using the existing literature in the field, thedocuments prepared for the launching of the monitoring project and resultsfrom the Intensive Training Workshop in Survey Methodology for the projectheld at UNESCO, February 1993, Paris, the author approaches theseinstruments with a learner-focused strategy that uncovers fundamentalquestions related to the assessment of literacy, numeracy and basic life skills,and the environmental factors influencing them.

This document serves to develop further the “think-tank” approach of thisInternational Monitoring Project. The results of this approach can only bereflected at the level of the national instruments developed by the participatingcountries. This document is strongly recommended in training workshops,courses and seminars for the development of educational survey instruments.

Vinayagum ChinapahDirector, Joint UNESCO-UNICEF Monitoring ProjectUNESCO

February 1993


CONTENTS

INTRODUCTION

Part I

TESTS

1. Test Construction1.1 Purpose and Approach: For What End Are We Testing?1.2 Selecting the Criterion Reference:

Learning Competencies to be Tested

LiteracyNumeracyLife Skills

1.3 Chosing Adequately : the Form of the Test and theFormat of the Test Items

1.4 Presenting and Sequencing the Items

Part II

QUESTIONNAIRES

2. Questionnaire Construction2.1 Purpose of the Questionnaires2.2 Content Areas2.3 The Questions2.4 Instrument Administration

CONCLUDING REMARKS

REFERENCE DOCUMENTS

3

34

4

51113

19

26

29

2929303135

37

39


INTRODUCTION

This reflection on instruments assessing learning achievement will cover pedagogical andlogistic aspects of instrument construction and administration.

The Monitoring Education for All Goals project identified learning achievement as onesignificant indicator embedded in the methodology of the project. Moreover indicatorsreferring to other factors influencing learning achievement and those referring to the issue ofaccess and equity are considered in the survey.

The term learning achievement refers to skills, knowledge and understaning that childrencontinuously acquire over the years of their lives. Formal schooling is considered a key factorin promoting this process of acquisition. More so primary education is supposed to lay thegroundwork for any further educational development, be it within or outside formalschooling. Ideally, at the end of their primary school cycle all children should have acquiredbasic learning competencies. The project therefore will look at learning achievement ofchildren after four years of schooling in the areas of numeracy literacy and life skills.

It is important to note, that the first and foremost objective of the project aims at thedevelopment of national capacities in monitoring their respectively defined education-for-allgoals in order to supply valuable information to decision-makers in the field of education.

In order to collect the necessary data on learning achievement, such instruments as tests andquestionnaires need to be developed. Not only is there the need to design them appropriatelyaccording to scientific criteria and culture specific contexts, but they also have to be carriedout competently.

The basic considerations given here about instruments assessing learning achievement, on theone hand reflect general findings on research methods in the social sciences and are of arather technical nature. On the other hand these general findings are considered within thespecific context of the Monitoring-Education-For-All Goals project, being an internationalproject with the objective of arriving at a common core of instruments on the grounds ofscientific methodology, but with special consideration of the country specific contexts.

During the project’s Intensive Training Workshop on Survey Methodology at UNESCO Parisin February 1993, where besides experts and researchers in the field of education the firstfive countries participating in the project were represented, the issue of instrumentconstruction was on the agenda. As the participants tried to reach consensus on a list ofinternationally relevant minimum competencies in the areas of literacy, numeracy and lifeskills, the discussion became lively. Whereas a common core of competencies concerningnumeracy was agreed upon without major controversy, identifying those involving literacyand life skills seemed a more difficult task.


The workshop group was international and consisted not only of educational researchers andexperts - most of whom had practical experience in educational monitoring, but also policy-makers from the different countries who had the chance to cooperate internationally. Someof the ideas, reservations and concerns mentioned there will be reflected in this paper.

Whereas the first part of the paper will deal with a range of aspects relevant to testconstruction, the focus of the second part will be on the collection of data involving aspectsinfluencing learning achievement utilizing questionnaires.

Issues such as the reasons for testing and more specifically the reasons for testing seen in thecontext of this project will be raised. This will be followed by a rather extensive reflectionon the concepts of literacy, numeracy and life skills leading to the presentation of the list ofexpected minimum basic learning competencies at the end of grade four as they were definedby the workshop group mentioned above.

Theoretical and methodological aspects of the actual item construction will follow. In parttwo of the document the relevance of collecting information influencing learning achievementwill be explained first. Further some methodological considerations of questionnaireconstruction within the context of the project will be presented. A few words aboutinstrument administration will be said at the end.

A brief conclusion at the end of the paper will point at the chances and limitations impliedin using these instruments.

2


PART I

TESTS

1. TEST CONSTRUCTION

Scientific test construction has to be approached systematically, if we want testingreliable, valid and feasible as possible. The following questions will be examined:

to be as

(a)

(b)

(c)

(d)

(e)

For what end and for whom are we testing?

What are the learning competencies to be tested and on what grounds are thechoices justified?

Which test type is most adequate for our purpose and what kind of questionsare appropriate or inappropriate - considering aspects like the age of the targetgroup, the financing, time, familiarity with types of questions ?

How are the items to be presented and sequenced?

Is the administration of the test feasible and administrable - considering theavailable institutional capacity, personnel or other relevant (country) specificfactors?

A careful analysis concerning the presentation and style of the questions is as important asthe actual content. The appropriateness of the relation between the target group and the testitems in terms of linguistic, psychological, developmental, and cultural aspects have to beconsidered carefully.

Test items have to be clear and choices given to answer a particular question must bemeaningful. Moreover the instructions given should be as clear and as motivating as possible.

1.1. Purpose and Approach: For What End Are We Testing ?

Since equality of educational opportunity has been at the centre of educational research, thecontroversy about testing and whether it is beneficial or inimical to the individual and tosociety (see Kellaghan et al 1982) has been discussed. It has been shown that testing bearsthe danger of labelling students or that test results are used to justify existing socialstratification (s. Kellaghan et al 1982). Therefore it is crucial to ask for what end the testingshould be utilized.

In the context of the Monitoring-Education-For-All Goals project the tests are used as aninstrument to gather data on how much a particular group of students know in relation to adefined body of knowledge. This data shall eventually be analysed in order to monitor the

3


education for all goals in a given country. The target group in this case is a sample of fourthgraders and the defined body of knowledge consists of minimal competencies in the area ofliteracy, numeracy and life skills (see 1.2.).

The approach described here is called criterion-referenced testing as opposed to norm-referenced testing. The latter approach is more explicitly comparative and interested inshowing how particular students or groups of students compare with a reference population.Bradburn and Gilford (1991) have demonstrated that the approaches are not mutuallyexclusive. It is possible to report norms also for criterion-referenced scales or to anchorcriterion-referenced scales; given that there are enough items and those items discriminate atvarious scale points (Bradburn & Gilford 1991: 7).

During the Intensive Training Workshop on Surveying Methodology the question ofinternational comparability of results was raised. It was emphasised several times, that theprimary interest of the project was not to compare, but to strengthen national capacities inmonitoring education-for-all goals as defined by the respective countries. Scaling the testresults in order to show where each country stands according to a defined international normdoes not seem to serve the expressed project objectives.

The task once again is to develop a prototype test on minimum but essential learningachievement in the areas of literacy, numeracy and life skills.

1.2. Selecting the Criterion Reference: Learning Competencies to beTested

“ But, since the conditions and needs vary from one country to another, andknowledge and technology are constantly changing, educational “standards” must beregarded as fundamentally “relative. “ (Beeby 1969 in Grisay and Mählck 1991; p.4)”

This quotation by Beeby on the relativity of educational standards reflects a spirit which isrelevant to this international project, but also to science in general. Since any finding isrelative in that it is related to certain assumptions and rules, there is no unquestionable truth.As the critical rationalist Popper has demonstrated, as scientists we are prisoners caught inour paradigms, expectations, past experiences and language (Popper 1963). As long as thescientist is aware of the necessary limitations concerning his or her work it is still desirableto progress in the attempt to describe social realities on the grounds of a given methodologyin order to give explanations and make prognosis.

One objective of this project lies in identifying a common core of minimum learningcompetencies which children after four years of primary schooling should have acquired. Aconsensus regarding basic learning competencies was reached among the first fiveparticipating countries at the Intensive Workshop in Survey Methodology at UNESCO inParis (February 1993). The consensus, which will be presented below, was based as muchon general findings in child psychology, child development, learning theory and curriculumdevelopment as on the ideas and judgments of the representatives present. Their soundknowledge of their culture of origin was crucial with regard to a need for an overall culturalappropriateness of the battery of learning competencies.

4


LITERACY

“ Literacy is probably the most important foundation stone for lifelong learning. “(Freeman 1992: 78)

International Literacy Year 1990 initiated by UNESCO was a viable tool for promoting theimportance of literacy for all. In numerous publications the importance and implications ofbeing literate have been explained and discussed. Generally, the more technically advanceda society is, the higher is the level of literary competence to take part actively - to function -in that society.

After a general reflection on the concept of literacy, the consensus reached among theparticipants at the Intensive Training Workshop in Survey Methodology will be presented.Here again the task was to decide on an internationally common core of minimum learningcompetence in relation to literacy, which should be acquired by pupils after approximatelyfour years of schooling.

Some Remarks on the Concept of Literacy

A basic definition of literacy is the ability to read and write. Freeman elaborates by arguingthat a literate person is able to present ideas using the written word, and understanding,storing and analyzing words and to react appropriately. It is indeed true, that once a personhas acquired the kind of competence described, she or he will be able to continue lifelonglearning (see Freeman 1992: 192). In their 1988 publication the UNESCO office of Statisticsstated that literacy is

I f. the application of a set of skills to a set of general knowledge areas, which result

from the cultural requirements that are imposed on the members of a culture. “(UNESCO 1988)

This statement emphasises the importance of cultural relativity of literacy when regarding theissue of different cultures. As mentioned above the more technically advanced a society is thehigher the level of literary competence to function in that society.

Word Knowledge

A prerequisite for reading is aWhereas it is essential for the

sufficient command of words and the way they are used.learner to have reached a certain level of linguistic and

communicative competence in the language used to acquire the reading skills, a learner witha higher level of linguistic and communicative competence usually progresses faster. Thereason lies in the fact that reading is all about giving meaning to a text. That meaning is

5


conveyed through signs and symbols which have to be familiar to the reader. In order to readthe learner has to understand that the written signs represent what she or he has experiencedas spoken language.

Reading

Deriving partly from ideas related to a child-centred learning and teaching approach, partlyfrom insights in receptionist theory the next section will discuss some thoughts about reading.

Reading is an interactive process involving the reader and the text and it is a translation ofsigns into meaning. The reading of a text will always vary according to the culturallyinfluenced and individually differing preconceptions of the reader. This fact has to be keptin mind when testing eleven year-olds from different cultural realities.

The extent to which a text allows for diverging interpretation also depends on the text-type.The reading of a descriptive text for example is more likely to lead to a consensus on themeaning among different readers than a literary text.

The reading of a literary text is a psychologically much more complex activity for the reader,because the subconscience is actively involved. The interaction between literary writings andthe reader can be most challenging since it demands negotiating abilities. In the process ofunderstanding (giving meaning to the text), the reader may have to go through feelings offear, of joy, of happiness and sadness, very close to what the child experiences and has tolearn to cope within real life. It is obvious that the reading of literary writing touches on theissue of life-skills, which shall be discussed later.

The above thoughts clearly show that reading involves more than the technique of identifyingletters and structures and that the testing of reading skills should reflect this complexity.

Writing

Writing involves more than reading and thus is more difficult. The learner has to understandletter-sound combinations and the concept of spelling.

In writing, the breaking up of words is physically reinforced by the movements of the hand.Children have to connect the words back together to put the meaning back, which is adifficult process and does not necessarily develop together with reading skills.

The physical difficulty often leads to frustration on the side of the learner, because she or hetranslates an idea into the signs.

Producing a piece of written work is intellectually complicated and involves keeping severalideas in mind at a time and working with them together in a narrative, while also attendingto the physical aspects of forming the letters legibly.

6


Mechanical Writing Skills

Some mechanical writing skills are essential for the production of the written word. Thereis the actual formation of letters, the ordering of letters into (correctly) spelled words alongwith the consideration of some grammatical features. These mechanical skills are easier toacquire when they are taught in a meaningful context.

Reaching Consensus

According to the participants within the scope of the international Monitoring-Education-For-All project a child after four years of schooling should be able to meet the following minimallearning objectives in the areas of vocabulary, reading comprehension and writing.

THE PROJECT’S COMMON CORE OF MINIMUM (ESSENTIAL) COMPETENCIESIN LITERACY

VOCABULARY The pupil should be able to ...

1. recognise written words2. identify synonyms versus antonyms

READING COMPREHENSION The pupil should be able to . . .

3. read and react to a simple (factual)question

4. read (comprehend) and react to shortwritten extracts presenting different texttypes

WRITING The pupil should be able to . . .

5. copy two short sentences6. react to a simple question in a written

form


Some highlights of the discussion

The Chinese representatives explained how acquiring literacy in Chinese differs from theother languages represented. The Chinese language consists of approximately 8000 characters,which all carry a different meaning. Experts in Chinese curriculum development have set thegoal for fourth grade primary school pupils to acquire 2500 Chinese characters by the endof the fourth year of schooling. Since the semantic variation depends to a large extent onintonation, the pronunciation and spoken language competence dominate the basic literacycurriculum. Therefore, the Chinese representatives were interested in including the skillreading out aloud as a minimum competence in literacy. Moreover, as a result of thecomplexity of characters it is crucial for a speaker of Chinese to be able to handle adictionary. This tool for the development of language competence is already introduced atprimary school level. Though these features will not be included as part of the common coretest items, it will be part of the country specific item in the Chinese test.

The issue whether or not to include defined competencies in the field of grammar wasdiscussed and supported by the francophone representatives. Grammar was not consideredrelevant for Chinese or Arabic speaking communities and therefore it was not considered toqualify as one of the common core of minimum learning competencies. Some countriesdecided to include it as a country specific relevant competence.

Another point raised referred to the competence of producing a written paragraph. Theparticipant emphasized the importance of the creative aspect of writing, irrespective forexample of grammatical accuracy or correct spelling. This point included the question ofwhether after four years of schooling this creative activity should be part of any student’scompetence and if the demand for good quality education in the sense to promote the child’sability to learn how to learn is to be met. Another participant questioned idea stating that hewould not consider that kind of exercise part of minimum learning, but rather a morecomplex exercise.

The author would like to make a comment on the point raised. It is true that from a learningpsychological point of view any activity which goes beyond recognizing and recallingknowledge can be considered more complex in that it may demand skills or abilities whichaccording to Bloom touch on levels of understanding, application, analysis, synthesis orevaluation which are considered higher in the taxonomy of learning objectives ( Bloom et al1972; also see Life Skills, below).

Other theorists have used other terms to develop a hierarchical description of learningachievement and analyse intellectual development. Guilford for example talks about thestructure of the intellect and identified five cognitive operations, such as memory,recognition, divergent production, convergent production and evaluation. He goes onto statethat divergent production is a prerequisite for creative behaviour (Guilford 1964). A thirdwell known concept is Piaget’s model of intellectual development where he claims that everychild goes through the same phases of intellectual development between the ages of 1 yearand 12 years and over time shows the capacity to master more and more complex operations(Piaget 1969).

8


These theories, which have - in their own right - promoted an understanding of how learningtakes place and how it can be promoted, have one weakness. They mainly deal with thecognitive domain instead of considering the learning process a holistic endeavour. There areindeed types of learners which find it extremely hard to master tasks which leave hardly anyspace for individual creativity, an activity equally involving the affective and psychomotoricdomain.

The same child may perform very well in a more creative task but not in a simple task andat the same time proof many of her or his other cognitive competencies which otherwisewould have remained invisible. If there is agreement that not only learning is a holisticendeavour but moreover that there are different types of learners (in a non-hierarchical sense)it may be justified to identify creative writing skills as one of the minimum learningcompetencies at the end of four years of schooling.

9


NUMERACY

The remarks on numeracy will be brief. There was not much controversy among theparticipants of the workshop as to what they conceived as basic numeracy (see below).

Some Remarks on the Concept of Numeracy

Numeracy is concerned with the pupil’s conceptual understanding, procedural knowledge andproblem solving abilities and therefore has great value for the intellectual development of achild.

Conceptual understanding refers to mathematical skills such as recognizing numbers, readingtables or figures, and comparing numbers. Procedural knowledge not only refers tomathematical skills that enable a child to add, subtract, multiply, and divide whole numbersand fractions but also to give meaning to decimal data. Problem solving skills which are oftenrelevant in daily life situations (money, length, weight, shapes, time) require conceptualunderstanding of numbers and numerical concepts.

Learning to use numbers sufficiently means that a child has to develop and co-ordinate threetypes of competencies. These include understanding how to solve problems, secondunderstanding why they can be solved in that way and thirdly deciding when a particulartechnique should be used.

Reaching Consensus

In the area of numeracy consensus was reached without any need for major discussion.Several participants mentioned how they felt more knowledgeable in the field of mathematicsas opposed to the other two areas, which may have been one reason for a quick consensus.

In many cultures, due to stigmatisation, especially women and girls have been affected bywhat Freeman called maths phobia - a mental block, which prevents people from learninganything which they perceive as mathematics, although, if they do not recognise it as such,they will not have any trouble with it (Freeman 1992: 85).

However, the list of minimum competencies in numeracy that will be expected within the testof this project consists of the following.

11


THE PROJECT’S COMMON CORE OF MINIMUM COMPETENCIES FORNUMERACY

NUMBERS

ARITHMETIC

GEOMETRY

DAILY LIFE

The pupil should know . . .

1. whole numbers and numerates

The pupil should be able to master simple . . .

2. addition / subtraction / multiplication / division

The pupil should be able to operate simple . . .

3. fractions / decimals / percentages

The pupil should be able to operate . . .

4. shapes and spatial relationships

The child should be able to solve

5. daily life problems of relevance to numeracyskills

Highlights of the Discussion

The representative from Morocco said that his country was interested in including in the testan item on computation. The other participants agreed to the importance of the issue, but feltthat it did not qualify as a minimum competence.

A Chinese representative mentioned that his country was going to include speed testing,which was not further discussed.

Clarifying where numeracy could be of importance to daily life problem-solving, the handlingof money and a sense of time were given as examples.

One participant articulated his concern about the three different fields literacy, numeracy andlife skills overlapping. Generally this was not felt to be problematic. The fact shows theinterdisciplinary nature among fields divided for scientific and theoretical purposes.

12


LIFE SKILLS

By including test items in the area of life skills the Monitoring-Education-For-All Goalsproject is covering relatively new grounds in the international field of educational monitoring.Not only is the concept of life skills in general difficult to define, but to decide on those lifeskills, which are meant to reflect part of the basic minimum competencies a child should haveacquired after about four years of schooling poses an especially difficult task. The discussionamong the participants at the above mentioned workshop shed some light on the culturalrelativity of what is essential to know for whom.

Before presenting the consensus reached on what should be included in the list of life skillscompetencies, a general background on this topic shall be presented. The concept of life skillsstill needs to be further elaborated and developed. The considerations made here may exceedthe Monitoring-Education-For-All Goals project in scope, but may inspire policy-makers andresearchers to become more innovative in this new field in the future.

A General Reflection on the Concept of Life Skills

The concept of life skills is not easy to define. To do so poses a rather complex andphilosophical task. Looking at the two parts of the term life skills, we will find that whereasit is possible to agree on an understanding of the meaning of skills, it is probably impossibleto arrive at a general consensus on what life means. According to UNESCO educational termsskills basically refers to abilities acquired by observation, study or experience in mentaland/or physical performance (UNESCO 1986). The term life implies a complex range ofmeaning including biological, psychological, philosophical or social aspects. However, inspite of knowing that due to the nature of things, the symbols we use in the form of languageto convey meaning will always be far from perfectly adequate, we should never refrain fromapproaching the complexity of the possible meaning carried by a word.

Philosophers, theologises, psychiatrists and presumably each one of us as a human being havesome contribution to make to the question of what life actually means. Whatever is being saidabout the meaning of life, the ideas will always be manyfold and determined by experiencesthat vary from person to person. A judgement on the question what the most important skillsare in order to manage life are subject to personal experience. We all are architects of ourown realities whether consciously or subconsciously and accordingly we interpret the meaningof life and the skills necessary to live. We are interacting with our environment constantlybeing confronted with and at the same time shaping the communities, situations,circumstances, ideas and beliefs around us.

13


Biophysical and Psychosocial Needs

The question of why we are the way we are and how determined we are due to our genes ordue to our environment are questions extensively discussed in the social sciences. We areborn with a theoretically immeasurable potential for development, but there are extensivelimitations to the development of our potential. Examples if these limitations could be aninsufficient satisfaction of basic biophysical or psychosocial human needs. A lack of cleanwater, malnutrition, the absence of shelter, for example, have a negative and even disastrouseffect on the biophysical development of a person, whereas such influences as under-expectation, stereotyping or rejection can restrict a sound psychosocial development. Thedivision between the two categories of human needs is purely theoretical and helps toconceptualize the life skills issue. It also touches on the question whether life skills shouldbe first and foremost understood as survival skills or whether they should go beyond that,leaving room for care and joy also. It is true though, that as long as the biophysical needsfor food, clean water, hygiene, health and shelter are not provided for, life willpredominantly be a struggle for survival.

Those life skills crucial for meeting the biophysical needs of a person demand knowledge onfacts of life which in their specific content will vary according to geographic and culturalcircumstances. This kind of knowledge is related to health and hygiene (i.e. immunization,diarrhoea, coughs and cold, hygiene, AIDS), nutrition or the environment (i.e. water supply,pollution). The joint publication Facts for Life by UNICEF, WHO and UNESCO is onesource which provides useful background information and knowledge relevant to these aspectsof the life skill issue. The information provided has already been adapted to the differentcultural contexts, but of course would have to be translated into test items for the purposeof this project. Since these aspects of life skills, which are more concrete in their nature andeasier to translate into test items, this general reflection will focus on the other, the so calledpsychosocial dimension of life skills.

As mentioned above, a child starts interacting with his or her environment from day one.This kind of interaction involves thinking and learning and is psychologically a highlycomplex process, which cannot be analyses sufficiently within the scope of this more generalreflection on life sills. The various schools of thought in child psychology differ, forexample, in their way of perceiving the baby child. Whereas Piaget builds his theory on theassumption that a child is absolutely egocentric at the beginning of her or his life (seeGinsburg & Opper 1991, p.130), Freeman believes in social interaction from day one(Freeman 1992, p.47). The theories also differ in their beliefs of how the mental developmentof a child takes place. Vigotski for example claims that mental development is dependent onlearning and that the child needs to be introduced to new areas to motivate his or herdevelopment through learning (Vigotsky 1978, p. 14). Piaget on the other hand came to theconclusion that mental development is not a result of learning, but that learning is a resultof development (s. Ginsburg & Opper 1991, p.212/213). Joan Freeman has discussed a rangeof different theoretical approaches to child development in her book Quality Basic Education:the Development of Competence which can be highly recommended.

The points relating to psychosocial aspects of life skills raised here reflect Freeman’s ideasand are supposed to broaden our awareness of some basic ideas relevant to defining theconcept of life skills.

14


In doing so some of the chances and limits of developing instruments for life skills testingwill become clearer.

The Immeasurable Potential

Whatever a child’s reality looks like and whatever reasons there are for the situation of aparticular child, he or she throughout life will have experiences, mature, think and learn.What these experiences will actually mean depends on several factors. Every child is bornwith a genetical potential which is so complex that it is impossible to be defined and whichallows for development to recognizable achievement. This process can be encouraged or alsodiscouraged by environmental factors, such as education.

Education can play an important role for the child by stimulating his or her psychosocialdevelopment in order to increase the child’s competence of making choices in life. If wepartly understand the concept of life skills as those abilities which enable children to makechoices, we have to look at when and how learning takes place.

Transfer of Learning

Learning for life takes place if what has been acquired by a pupil in one situation can betransferred to another appropriate one. This process proves that the pupil is able to abstractthe key operations from a task and recognises how they can be used in another. These tasksmay be similar, like calculations in maths and in physics or superficially different such aslanguage and craft. Helping the pupil to see similarities between problems and responses isessential to learning in any situation and can eventually lead to a monitoring competence ofone’s own learning.

Perceptional Learning

What are the necessary skills then to see similarities between problems and responses and toachieve conscious learning competence? How does this competence develop? Freemanexplains that intellectual skills such as skills to cope with incoming information, to store itin flexible categories in the memory, to retrieve it for application to different situations, andto adapt new information are necessary for learning. These specific skills have to be refinedto reach levels of competence. This needs directed teaching, such as distinguishing betweenshapes, recognizing forms and estimating distances. Through our senses we are able to learn.Our senses allow us to actively seek out for what interests us. We see, taste, smell, touch andhear. This kind of perceptual learning based on experience, involving skills such as attention,identifying and locating helps to explain how already a young child can acquire a languageor can learn how to walk (Freeman 1992, p.47). Perceptions may sometimes prove to be amisconception in as far as instead of enhancing the process of acquiring competence, they slowit down. Therefore the quality of education should be defined according to these insights toskill training.

15


Good quality education is largely concerned with correcting misconceptionsby broadening a child’s view in such a way, that the child can reach personalconclusions j-em what he or she has observed (Freeman 1992; p.49).

Social Skills

Social skills are another important skill which certainly belongs to the life skills category.Social skills - how to get on with people - involves knowledge of the self and others. Achild’s social cognition will greatly influence and at the same time reflect her or hisbehaviour. Social cognition refers to the way in which an individual perceives other peopleand comes to understand their thoughts, emotions, intentions and view-points. There isevidence, however, that even a complex and differentiated social cognition of a child does notnecessarily mean that he or she acts in a pro-social way. Moreover, children who are veryaware of the possibility to act voluntarily when helping, sharing, reacting to distress orshowing physical affection for the benefit of others without expecting reward, do notnecessarily behave according to that knowledge. There is a discrepancy between the cognitiveactivity and the affective or to put it simply, between the head and the heart. The reasons canbe manyfold, and often they are a reflection of a painful experience of the past.

Summary of Psychosocial Life Skills

The following list summarizes some central aspects of those skills that fit the category ofpsychosocial life skills. It would be interesting to discuss if or in how far they may beconsidered universally relevant. The list closely reflects some of the points Freeman raisedin her book on the quality of basic education (Freeman 1992).

(i)

(ii)

every-day problem solving skills or creative thinking

alternative solution thinking - a technique of trying to get somewherewhen one’s idea is blocked by trying to find an alternative solutionconsequential thinking - this involves thinking through theconsequences of one’s own or any kind of actionmeans-end thinking / reasoning - this involves differentiated planningin order to reach a certain goal and to foresee possibly occurringobstacles

social skills

how to get on with people; this involves social cognition attempting tobroaden one’s own perception of others and communicative competencehow to care for other people

16


(iii)

(iv)

thinking competence

thinking about thinking - an awareness of one’s intellectual assets, suchas thought, process, concentration and memorymonitoring one’s own comprehension and perceptions

conceptualizing skills

differentiating between concrete and abstract concepts

This list is only an attempt to approach the complex issue of life skills and is by no meanscomplete.

The Chances and Limitations to Testing Life Skills

One question raised above still remains to be answered. The chances and limits of testing lifeskills shall be articulated next. Bloom’s distinction between three domains of learningprovides a useful concept for discussing the question of measuring 1ife skills achievement.According to Bloom (Bloom (1956) 1972) there are three domains which need to beconsidered within teaching: the cognitive, the affective and the psychomotoric domains. Thecognitive domain includes those objectives which deal with the recall or recognition ofknowledge and development of intellectual abilities and skills.

Within his taxonomy of six cognitive learning objectives Bloom distinguishes different levels.He considers the first of them as a rather simple level of a learning, namely the level ofknowledge. The other five levels are considered to be more sophisticated and refer tounderstanding, application, analysis, synthesis and judgment/assessment. Along with thecognitive domain he identified the affective and psychomotoric domain. Whereas the affectivedomain deals with those objectives which describe changes in interest, attitudes and valuesand the development of appreciations and adequate adjustment, the psychomotoric domainincludes those objectives related to the manipulative or motoric skills area. Very oftenteaching proves to put too much emphasis on the cognitive domain while neglecting the otherdomains which may result in imbalances within a child. Bloom’s theory is one attempt toprovide a model to gain a better understanding of what learning and teaching means, but itbears the danger of leading us to perceive human beings too rigorously according to thesecategories. It is important to note that all the aspects reflected in Bloom’s terminology areconstantly in action and that in reality one cannot separate the different domains as neatly asin the theoretical concept.

Coming back to the possibility of testing life skills achievement, we will find that it is mostfeasible to test competencies within what Bloom considers the cognitive domain, lesser sowithin the psychomotoric domain and probably hardly within the affective domain. At thesame time, whether intended or not, most of the times all three domains are always involvedin testing, since learning achievement is a holistic endeavour. Still, the subject of assessmentrests predominantly within the cognitive domain. If one wanted to test psychosocial life skillsinnovative testing methods would have to be developed. The methodology would probablyhave to be different and the instruments would presumably involve sketch performance,observation activities, real life situations or group activities.

17


Reaching Consensus

All participants emphasized their deep interest in the subject of life skills. Due to timeconstraints the concept was not approached in its total substance. However, the consensusreached in the end gives some indication that the participants were content. Those minimumcompetencies selected as part of the common core shall now be presented.

THE PROJECT’S COMMON CORE OF MINIMUM (ESSENTIAL) COMPETENCIESIN LIFE SKILLS

LIFE SKILLS The pupils should . . .

1. know how to act when issues of health are concerned(preventative and remedial)

2. know how to act when questions of hygiene are concerned

3. know important facts concerning his or her nutrition

4. know about the environment around her or him

5. have some knowledge about skills of productive/pre-vocationalnature

6. be able to solve problems of daily life (social-civic, conflict-resolution)

Highlights of the Discussion

One of the participants raised the question whether or not achievement in the field of lifeskills could be considered an output of schooling. He felt that these competencies wereprobably acquired elsewhere. The statement was supported by another participant who saidthat as opposed to the formal school system, which apparently often neglects the issue of lifeskills, the informal educational delivery systems were the place to look for successful lifeskills teaching or empowerment. Other participants did not see so much of a problem, sincethey felt that the results of the achievement testing within the Monitoring-Education-For-AllGoals project could probably shed some light on the issue, which ultimately could be usefulfor educational policy-making.

The participants became very aware how important the cultural specific context is when itcomes to life-skills. What is a dangerous disease in one place such as malaria in Mali, maybe absolutely irrelevant in another area such as Jordan. Ideas about health and nutrition vary.

18


Milk products for example, often considered an essential part of a healthy diet in manycountries, are hardly consumed in China.

Conflict resolution strategies also vary according to specific cultural contexts. In somecultures people are much more confrontational than in others. Often the behaviourial normof conflict behaviour differs according to a persons gender.

One participant reported that his country was very interested in life skills as a means of self-advancement, involving learning strategies and information gathering. It could be a challengeto develop this idea further.

As mentioned previously, much more could have been said about this important issue.International education for example and its relevance for the area of life skills would probablyhave evoked an interesting discussion. But as so often, time ran out. Anyhow, one has to beaware that the project is approaching a comparatively new field and will certainly beinnovative to some extent. Hopefully the monitoring of educational goals with regard to lifeskills will build on the findings of this project in the future. In life skill teaching there is agreat potential for promoting a better understanding of individuals within themselves andamong one another, and also for promoting international understanding. These are ideas forthe future. As far as this project is concerned the easily reached consensus on a common coreof life skills issues is promising way.

1.3. Choosing Adequately : the Form of the Test and the Format of theTest Items

The objective of the points raised here shall increase the sensitivity towards the multi-facetedproblems that occur when designing test items in general and with regard to this study beingan international activity.

The Monitoring-Education-For-All Goals project has been very clear about its approach thatwhile following a common methodology and framework the country specific needs arefollowed on all levels. For the instrument construction this implies that not only the tests, butalso the questionnaires will be devised at the national level.

“ A proposed international study has to should display sensitivity to the culturalcontexts (e. g., language spoken, religion, laws, implements used, values held) for theeducation dimension to be assessed “ (Bradburn / Gilford 1992: 20].

Apart from this projects first and foremost objectives to increase national capacity building,the methodology of test construction will also show, why it is so important to have therespective countries themselves designing the items, if the test wants to claim the maximumpossible relevance and validity.

The guiding common core achievement criterion according to which the test shall be designedhas been discussed. Now other aspects of test construction of a scientific and technical natureneed to be considered.

19


THE TYPE OF TEST

There are a few essential points to be acknowledged whenever devising a test. They refer tothe validity, reliability, objectivity, feasibility and administrability of the test and have a sortof guiding function.

(a) Validity

A test should measure what it is intended to measure.

Maximum precision when defining the subject under test is a helpful if notessential exercise before operationalizing subject into a test item.The validity can be judged by checking the correspondence between the itemsand the defined content criterion (content validity). It may prove helpful tocontrol the validity of an item by asking if the questions really search foranswers in the subject under test, or if it is general knowledge, for instance,a large factor in the expected replies? Therefore, the subject under test has tobe as precisely defined as possible.

(b) Reliability

A test should produce much the same result if given again to the same pupilwithin a short space of time and irrespective of the person correcting it - itshould be consistent.

Individuals naturally may vary to some extent taking the same test at differenttimes, but for a test to be reliable the acceptability of inconsistently occurringerrors has to be limited

Whether a test can be totally reliable, producing the same scores over and overagain, and yet not measure what it was intended to, that is, not to be valid, atest which is valid is always reliable.

(c) Objectivity

The marking of a so-called objective test should ideally not deferrer accordingto the person marking it.

Test items which require short and ready answers are most likely to meet thiscriterion. Interestingly enough experience has shown that when essay-typeanswers are marked on specific points, the final marks can vary considerablybetween teachers, but a mark from a subjective impression of the work hassurprisingly often not been very different.

20


(d) Feasibility

The test should bedefined time frame

carried out and assessed within a more or less preciselyand budget plan.

Time and money are both factors which have an impact on the possiblyfeasible scope of a testing endeavour.

(e) Administrability

The test has toinstitutional, andAdministration).

be administrable according to available personnel,financial capacities (see 3. Questionnaire and Test

The test type regarded as guaranteeing the mentioned criteria best is a paper-and-pencilstandardized test. There is growing concern that in these tests too much emphasis is put ontesting pupils’ achievement within the cognitive domain, while other domains of importanceare neglected. Since we are interested in developing indicators of learning achievement, whicheventually allow us to draw conclusions for improving the quality of basic education ingeneral, more emphasis will have to be put on assessing manual skills and behaviourialpatterns (see Ross & Mählck 1990). In this context Bradburn and Guilford refer toperformance testing as a promising instrument to test manual and behaviourial skills. Theymention the progress that has been made in developing them. Coming back to the abovecriterion of a test’s feasibility it has to be said though, that performance tests are moreexpensive than achievement tests and pose more logistical demands on test administrators.Since the Monitoring-Education-For-All project aims at the development of simple tests, apaper-and-pencil achievement test seems most appropriate.

Types of Items

Constructing a test item is a complex task.. Figure 1. shows the different aspects which needconsideration. The questions whom and what to test have been discussed above, but now needto be considered again within the item construction procedure. In the fourth column themedium of testing as an essential part of the item is mentioned. At the bottom it says thatthere is need for adapting the medium of testing to the cultural specific context. This basicallymeans that the medium has to make sense to the tested pupil. In other words, only if themessage conveyed through the medium of testing is meaningful to the pupil will the itemmeet the criterion of validity.

What is meaningful to a pupil has been largely shaped by her or his cultural specificexperience. If for example a familiarity with the concept of testing exists, this is probably dueto personal experience within that country’s particular schooling system(s). What themes,what pictures, what language or what type of questions are appropriate depends to a largeextent on the cultural context and general environment in which the test is carried out.

21


As the following figure shows the appropriateness of a medium also depends on the subjectof testing.

TEST ITEM CONSTRUCTIONwithin the framework of an international project

WHOM to test? WHAT to test? WHERE to test? HOW to test?

TARGET GROUP SUBJECTto be tested of testing

PUPILS AFTER MINIMUMABOUT FOUR COMPETENCEYEARS OF (LITERACYSCHOOLING NUMERACY ANDIN DIFFERENT LIFE SKILLS)COUNTRIES

LOCATION MEDIUMof testing for item construction

COUNTRY QUESTIONSRURAL/ EXTRACTSURBAN AND PICTURESTYPE OF NUMBERSSCHOOL THEMES

CONCEPTSLANGUAGE

HOLDERS OF DEFINED DIFFERENT NEED FOR ADAPTINGDIFFERENT COMMONCORE ENVIRON- ACCORDING TOCULTURAL MENTS CULTURALSPECIFICIDENTITIES CONTEXT

Figure 1. Considerations for Test Item Construction in an international survey.

In the following paragraph a few words about what medium to chose for the different itemsshall be said. Please note, all aspects of the defined common core of minimum learningcompetencies, are not covered.

Types of Questions / Items

Irrespective of the type of question, they should avoid ambiguity and stereo-typing.

Open-ended Questions

One can basically distinguish between open-ended and closed-ended questions (seeQuestionnaires below). During the discussion at the Intensive Training Workshop onSurvey Methodology with the exception of one item the latter type was consideredmore appropriate for the project’s purposes. Using open-ended questions affords

22


writing skills, which in themselves are subject to testing in this project. The validityof the item would not be guaranteed.

The open-ended question was used only once in an item on writing skills (literacy)achievement. The pupil was asked to look at a picture and to imagine what wouldhappen next. This essay-type answer involved reasoning and selection and gave roomfor creative thinking.

Closed-ended Questions

Closed-ended questions are usually easier and quicker to master. For two reasons theiruse is also somewhat problematic though.

(a) Pupils tend to guess what may be the correct answer among a given choice.

(b) Moreover it is difficult for the test designer to always find incorrect answers,which at the same time are somehow meaningful in the given context. In theexample below the person constructing the item did not succeed in findingmeaningful options.

(i) Imagine that you have been suffering from continuous diarrhoea foralmost two days. What will you do?

I will go and see

A. a policeman

B. my friend

c. a nurse, a doctor or my mother

D. a film or cinema

Choice D. is not meaningful in this context. Moreover the answers are neitherclear nor precise. A child choosing to see her or his mother in this situationwould not know whether or not to circle C., since there are other peoplementioned in the same answer, whom he or she would not see.

True-False Items

A statementdon't know

is presented and to be judged either true or false. Sometimes Iis given as an option. During the workshop this option was

discussed and considered for the item on nutrition. The statements would haveto be adapted to cultural contexts.

23


(ii) A. Drinking plenty of clean water is good for our bodyB. It is healthy to eat fishC. It is good to eat three times a day

TRUE FALSE I DO NOT KNOW

CORRECT- WRONG items aiming at identifying the formal correctness of astatement or SAME - DIFFERENT items follow the same pattern.

Multiple Choice

Multiple Choice tests give a choice of answers from which to choose.Sometimes a question is explicitly being asked and a choice of answers given(see above), sometimes The beginning of a sentence is presented and theappropriate finishing part has to be found. Below is an example discussed atthe workshop. In the literacy part of the test instructions on how to quantifyvitamin tablets was presented and the following form of multiple choice given.

(iii) Each tablet contains . . .

A. vitamin EB. vitamin Kc . vitamin PD. vitamin A

It is again not an example where as they should all answers are meaningful.

Correct Matching

Matching involves recognizing a common feature according to which matchingis justified. In the set of items proposed for discussion at the workshop therewas a word and picture test where the child had to match a word with theaccording picture. Participants agreed to take up the item in their set of testsadapting the choice of vocabulary and pictures to their specific context.Subject of testing in this item was word recognition.

Time

Appropriateness of time estimation refers to the concentration span that can beexpected from the sampled students according to their age. These may varyfrom culture to culture, but experience has shown that a whole test should notexceed the time span of about an hour. The IEA study on reading literacyallowed breaks in their test design. There is a danger confusion which may bedifficult to handle.

24


At the intensive Training Workshop on Survey Methodology the representativefrom Jordan remarked that they were not going to give all three sets of teststo every pupil, but sample enough students in order to give just one of thesets to each student. That procedure prevents the testing (a) to ask the pupilfor more concentration than he or she realistically has and (b) to take too muchfrom the regular teaching time.

Familiarity

If the sampled students are not familiar with the kind of testing to beundertaken, they may not be able to master the test. In that case lowachievement results will not necessarily reflect where a student standsaccording to the subject under testing, but instead may reflect the fact that astudent is confused about the logic behind the test and is still in the processof understanding the procedure (see (a) validity).

Pilot-testing

New or substantially revised tests should be pilot-tested to ensure the qualityof individual items and instructions to exams, as well as the appropriatenessof time limits of the questionnaire (Bradburn / Gilford 1992: 28)

The results of the pilot-testing should be carefully analyses for any kind ofitem bias or inappropriateness.

Some Guiding Questions for Item Construction

The following questions may serve a test constructor in any of the countries as auseful guide.

(1)

(2)

(3)

(4)

(5)

What is the subject of testing?

Which type of question/item seems appropriate for measuring this subject andon what grounds can the choice be justified (pedagogical findings, teachingexperience, psychology of learning, child psychology, didactic considerationsetc.)?

Is the item clear and specific?

Is the target group familiar with the medium used in the item?

Is the language use appropriate for the target group?

25


(6) Does the theme chosen consider the age and the cultural context of the pupils?

(7) Are you able to predict and justify the time it will take to complete each item?

(8) Is there any need for changes after the pilot-testing results have been analyses?

1.4. Presenting and Sequencing the Items

A desirable presentation of test items concernsinstructions given, both

Presentation

The presentation

of which should be clear

andsenses are somewhatshould not only havemisunderstanding the

(1) Leave enough

their visual appearance andand motivating as possible.

the relevant

layout will influence the participants performance. All of ourinvolved in what we doa motivating effect, butformat.

space between the items

and the visual presentation of a testalso help to prevent mistakes due to

so that it is visually clear where oneitem starts and where it ends.

(2) Present what ever is necessary to respond to a question in “an appropriatemanner. If the task is to circle a letter indicating the correct answer, makecertain the letter is present and in a position that it is easily associated with theparticular question. To write a sentence the space for it has to be appropriate.

Instructions

However well a specific question has been devised, if the instructions along with itare inappropriate, it may remain unanswered. The following list of recommendationscan be used as a checklist for presenting the instructions.

(1) You want to get the tested person’s attention. Therefore address him or herdirectly.

(2) The initial instructions for the test should explain the reason for testing andmotivate the participant to take part. The tone of language can often make adifference.

(3) Try to give a context for the task to be mastered which helps in focusing onthe particular question.

26


(4) Be simple and as clear as you feel is possible, knowing that an element ofsubjectivity cannot be absolutely avoided.

(5) The language use should always be correct.

(6) Be polite in addressing the pupils.

The way the test administrator introduces the items is already part of givinginstructions. His or her tone, personality and mood will influence the pupils attitudetowards the test and their motivation .

Sequencing

The didactic rule is to progress according to the difficulty of a task. For the literacyitems chosen this would mean to progress from word-picture recognition to producinga few free sentences in a written form. In numeracy the pupil will be first tested ona general understanding of numbers before applying the knowledge for problemsolving. Concerning the life skills items the factual questions are more straightforwardto answer than the items on problem-solving in daily life situations.

At the workshop the participants supported different ideas as to what extent thetaxonomy of learning objectives presented in the items should reflect different stagesof primary schooling. One participant strongly supported the idea of including slowlyprogressing items to see where the particular pupils stand according to the definedexpected minimum of achievement. This information may for example allow us to seecorrelations between the type of learning achievement and for example the teachingmethods. In the end it was agreed that the questions should be of varying difficulty.

27


Part II

QUESTIONNAIRES

2. QUESTIONNAIRE CONSTRUCTION

The set of questionnaires are instruments to gather information on other influences to achild’s achievement than the school alone. In the early 1960 a lively controversy dominatedthe educational science arena evolving around the question to what extent the homeenvironment as opposed to the school environment accounted for a child’s scholasticperformance. Around the same time the decisive role of socio-psychological, cultural andlinguistic factors on cognitive growth and development started to be examined in depth (seealso 1.2. above).

The insights of an environmental social-psychological approach were of supreme value forconceptualizing multiple environment influences on school learning (see Bloom 1964;Chinapah 1983). Chinapah in his model of school learning emphasised the importance ofprocess variables for a sufficient analysis of learning achievement. These process variablesrelate for example to parental educational support or to pupil-teacher interaction. Relevantglobal characteristics on the other hand give information of a different kind such as theavailability of books in the home, parental socioeconomic status, ethnic belonging or the ageor gender of a pupil,

2.1. Purpose of

to name just a few.

the Questionnaires

Therefore the set of questionnaires serves as an instrument to gather information on thosemultiple environment influences relevant to the achievement of the grade IV target grouppupils. Cultural sensitivity is required to find not only the right tone in addressing therespondents, but also in deciding whom to ask about what.

The following sets

1. The

2. The

3. The

of questionnaires shall be designed to gather the relevant information:

pupil questionnaire on the student’s home and school environment.

parent questionnaire also on aspects of the student’s home environment.

teacher questionnaire on the teacher’s background, instructional practicesand beliefs. -

4. The school headmaster questionnaire on the school settings, policies andpractice.

29


2.2. Content Areas

Since the areas relevant to the student’s scholastic performance have been identified as (1)the student’s home and school environment, (2) the teacher’s background, instructionalpractices and beliefs, (3) the school settings and policies, a decision on the choice of variablesfor the sets of questionnaires has to be made.

In this process several aspects relevant to the choice and the length of the questionnairewithin the context of this particular project Monitoring-Education-For-All Goals need to beconsidered.

(a) The questionnaires should be kept as short as possible, while assuring that thenecessary information is gathered. The choices have to be justified onappropriate grounds.

(b) Only those variables should be chosen, that serve to gather information thatcannot be collected more easily by using other sources of information (schoolstatistics, teacher’s and pupil’s records, administrative information).

(c) The cultural specific context of the countries has to be considered and may1ead to variations in the choice of variables.

The Monitoring-Education-For-All Goals project outline lists the followingcharacteristics and variables.

Student Characteristics: age, sex, health, nutrition, height, weight, socio-economicbackground, cultural-linguistic status, etc.

The School Setting: teaching-learning conditions (teacher characteristics, availabilityof textbooks, school library, school facilities, school feeding, sanitary services, classsizes, budget per student etc.)

The Community Environment: school-community relationships (communitycontributions in cash and kind, functioning of parents-teacher association, etc. )

30


2.3. The Questions

At the heart of any questionnaire are theresearch objectives into specific questions.

questions. The questionnaire must translate theThese questions should motivate the respondent

so that the necessary information can be gathered. As has been discussed in the context ofitem construction, clear phrasing of the questions is essential, if we want to avoidmisunderstandings. When formulating a question its content, structure, format and sequencehave to be considered.

Formulating the Questions

Devising a question demands carefully designed and clear phrasing. The response setsshould always show complete sentences.

(a) Wording of the Questions

The choice of wording in a question should depend linguistic competence of the targetgroup.

(b) Leading Questions

It has to be kept in mind that the phrasing of a question influences the respondentstheir answers. The term leading question for example refers to a question phrasedsuch a manner as if the researcher expected a certain answer.

inin

Example: A question about a student’s involvement in housework might read,“How do you feel about the house work you do at home?” The samequestion phrased in a leading form might read, “Do you think that youhave too much house work to do when you are at home?”

Questions should neither support accepted norms nor reflect socially undesirablebehaviour, since the respondent is likely to respond according to the sociallydesirable.

Example: Do you try your best in your homework?

(c) Presenting the Questions

The presentation of a question has an influence on the respondent’s perception andmotivation. Directly addressing the respondent for example, decreases the distancebetween the questionnaire and the respondent and

Example: Do you have any brothers/sisters?

has a motivating effect.

31


Content of Questions

(a) Factual Questions

Factual questions are designed to elicit objective information from therespondents regarding their background, their environment, their habits, andthe like.

Example: How old are you?

Other kinds of factual questions are intended to provide information on therespondent’s social environment.

Example: How do you get to school?

(b) Opinion Questions

Opinions are the verbal expression of attitudes. The concept of attitude refersto the sum total of a person’s inclinations, prejudices, ideas, fears, andconvictions about any specific topic.

Example: How many more years would you like your child to attend school?

Types of Questions

(a) Closed-ended Questions

In a closed-ended question respondents are offered a set of answers fromwhich they are asked to choose the one that most closely represents theirviews.

Example: About how many books are there in your home?

None or a few (l-l0)

Some (11-30)

About one shelf (31-100)

More than one shelf (more than 100)

32


(b) Open-ended Questions

Open-ended questions are not followed by any kind of specified choice, andthe respondent’s answer is recorded in full. There is a lot to be said for posingopen-ended questions, but the practical difficulties and the cost of scoringinvolved in analyzing them have to be considered carefully.

Example: What would you like to do in the future?

Question Format

(a) The General Format

The general format is to present all possible answers and have the respondentcheck the appropriate categories of closed-ended questions. The respondent caneither circle the answer or check a box or a blank. Specific directions shouldbe provided as whether the respondent is to circle a number or check a blankor a box. The method of circling a code number or the box method isrecommended, since experience with the blank method has shown, that therespondent will check between the blanks and it will be difficult to tell whichcategory was intended.

(b) The Rating Scale

A common and useful format for our purposes is the rating scale. Therespondents are asked to make a judgement in terms of sets of orderedcategories, such as “very good”, “good” or “average” “not good”.

Example: How good are you at reading?

not good 1average 2good 3very good 4

(c) Matrix Questions

The matrix question is a method for organizing a large set of rating questionsthat have the same response categories.

33


Example: Which of the following types of assessment do you (1) frequently use,(2) occasionally use, (3) rarely use, (4) never use

Frequently Occasionally Rarely Never

Standardized tests ( ) ( ) ( ) ( )

Performanceon homework assignments ( ) ( ) ( ) ( )

Performanceon classroom questions ( ) ( ) ( ) ( )

Sequence of Questions

After having determined the content and the format of the questions, we have toconsider the order in which the questions are placed in the questionnaires. Questionsthat are presented first in the questionnaire should put the respondent at ease andtherefore these opening questions should be easy to answer and not deal with difficultand sensitive issues.

With regard to the two types of content questions, the factual questions shouldprecede the attitudinal questions. Accordingly, the closed-ended questions should beposed before the open-ended, since the answering of the latter type usually requiresmore time and thought and may reduce the respondents’ initial motivation tocooperate.

Highlights of the workshop discussion

One general problem arising when designing the questionnaires was discussed at the IntensiveTraining Workshop on Survey Methodology in Paris and concerned the decision of whom toask what question. One participant was doubtful as to how reliable the pupil’s answers weregoing to be. As an example he gave the question on the distance between a pupil’s school andhome. The representative from Mali was especially concerned about the ability of someparents to answer a written questionnaire, since many of them are illiterate. It was decidedthat there is no clear solution to the problem, but that countries should make the relevantdecisions as to where to get the different data on the bases of their country’s specificsituation. It was felt that possibly in some countries a survey administrator would need to goto the parents, even if such an interview like situation may have an impact on the answers.Past experience had shown that in some cases the teacher assisted in the communicationprocess.

34


One other representative saw a potential problem in that questionnaires addressed to parentsvery often were filled in by the father only, though the answers could look different if it wasthe mother answering.

A representative from The People’s Republic of China did not see how the parent’s attitudetowards education could be of importance to their child. Other participants supported theimportance, claiming that a positive attitude on education on the side of the parents wouldbe felt by the child and have an encouraging effect on his or her attitude towards learning.

2.4. Instrument Administration

To assure a sound administration of testing and utilizing questionnaires, it is necessary thatadministrative procedures be controlled and as nearly identical as possible. Standardizedprocedures for instructing the pupils are indispensable, if the test results are to be as objectiveas possible. Therefore precise and detailed manuals on how to administer the test should beprovided. Variations according to specific regional or country needs have to be respected, butexplicitly mentioned in the survey report. It is desirable if not crucial to train the assistantadministrators. Everybody involved in the testing has to know what is going on and why.

With regard to the questionnaires the parent questionnaire administration may pose problems,since pupils are likely to forget to give them to their parents or because the parents areilliterate. Teachers have sometimes taken a mediating role in the procedures.

35


CONCLUDING REMARKS

A whole array of considerations on instruments for assessing learning achievement rangingfrom critical pedagogical and socio-psychological to methodological and technical aspectshave been presented in this document.

It has been shown how in the context of an international project like the Monitoring-Education-For-All-Goals project it is possible to take a common methodological approachtowards instruments construction while at the same time allowing for country specificadaption.

The paper has given an extensive reflection on the concepts of literacy, numeracy andespecially on life skills. Referring to literature on child psychology and learning theory whichpromote a child centred, interactive and holistic learning approach, some limitations ofstandardized testing as a form of assessment have been depicted. It was explained how muchthese tests are geared towards the cognitive domain which poses problems when suchcompetencies as creativity and behaviourial skills were to be assessed.

These limitations should be born in mind and possibly be reconsidered for furtherimprovement of instruments for assessing learning achievement in the future. Especially theconceptualisation of life skills could be further elaborated, considering for example theimportance of international education and inter- / multicultural competence.

At the same time the criterion for testing as outlined in the lists of minimum competenciesresulting from the Intensive Training Workshop in Survey Methodology at UNESCO in Paris(February 1993) can be justified according to the defined scope of this particular survey,which asks for simple tests in order to be administrable. The survey will be one importantmilestone on the way of monitoring education-for-all goals covering new grounds in the areaof life skills testing.

The document has moreover provided the reader with valuable theoretical and practicalinformation on test and questionnaire construction to assure their scientifically definedvalidity, reliability and maximum possible objectivity.

37


Reference Documents

Audouin-Leroy, C. ,Fomba, C.-O- & Hajji, N. (1992): Prototypes de questionnaires,réflexions de méthode. Institut de Recherche sur L’Economic de L’EducationUniversité de Dijon, France. Projet “Monitoring” Education Pour Tous. Etudes etdocuments de travail No 3, Division of Basic Education, UNESCO Paris.

Beeby, C.E. (ED.). (1969): Qualitative Aspects of Educational Planning. IIEP, Paris.

Becker, G. E. (1987): Planung von Unterricht.Handlungsorientierte Didaktik Teil I. BELTZVerlag. Weinheirn und Basel.

Bloom, B.S. (1972 (1956): Taxonomie von Lernzielen im kognitiven Bereich. Weinheim undBasel.

Bradburn and Gilford (1991): A Framework and Principles for International ComparativeStudies in Education. BICSE, National Research Council, Washington.

Chinapah, V. 1983: Participation and Performance. Institute of International Education.University of Stockholm, Sweden

Chinapah, V. / Miron, Gary (1990): Evaluating Educational. programmes and Projects:Holistic and Practical Considerations. Socio-economic studies 15, UNESCO, Paris.

Chinapah, V. (1992): Monitoring and Surveying Learning Achievements. A Status report.Monitoring-Education-For-All Goals Project. Studies and Working Documents No 1,Division of Basic Education, UNESCO, Paris.

Cook, S. W. / Wrightsman, L. S. / Stelltiz, C. (1976): Research Methods in Social Relations- Third Edition. HoIt, Rinehart and Winston, United States of America.

Elley, W. B. (1992): How in the World do Students Read? IEA, Grindeldruck GMBH,Hamburg.

Freeman, Joan (1992): Quality Basic Education: The Development of Competence. UNESCO1992.

Ginsburg, H. & Opper, O. (1991; 6. Aufl. ): Piagets Theorie der geistigen Entwicklung. Klett-Cotta / Psychologies, Stuttgart.

Grisay, A. & Mählck, L. (1991): Issues and Methodologies in Educational Development: anIIEP Series for Orientation and Training. No. 3 The Quality of Education inDeveloping Countries: A Review of Some Research Studies and Policy Documents.IIEP, UNESCO, Paris.

3 9


Kellaghan et al (1982): The Effects of Standardised Testing. Library of Congress. Boston/TheHague/London.

Lockheed, M. E. & Komenan, A. (1988): School Effects on Student Achievement in Nigeriaand Swaziland. Human Resources Department. The World Bank, Washington.

Mauritius Examinations Syndicate (1992): Learning Competencies for All. MauritiusExaminations Syndicate.

Nachmias, Chava & Nachmias, David (1991): Research Methods in the Social Sciences -Alternate and edition. St. Martin’s Press, Great Britain.

Pawar, P., Daun, H. et al (1992): Draft Instruments for Assessing Learning Achievements:Survey Questionnaires and Tests. Institute of International Education, StockholmUniversity, Sweden. Monitoring Education-For-All Goals Project. Studies andWorking Documents No.2, Division of Basic Education, UNESCO, Paris.

Popper, Karl (1963): Conjectures and Refutations. New York: Harper & Row.

Ross, K.N. & Postlethwaite, T.N. (1991): Indicators of the Quality of Education. A NationalStudy of Primary Schools in Zimbabwe. UNESCO/International Institute forEducational Planning, Paris.

UNESCO (1986): Glossary of Educational Technology Terms. IBE, Data/ UNESCO. Geneva.

UNESCO (1988): Compendium of Statistics on Literacy. Office of Statistics, UNESCO, Paris.

Vigotsky, L.S. (1978): Mind in Society. The Development of Higher Psychological Process.Cambridge, MA, MIT Press.

Wendeler, J. (1969): Standardarbeiten. Weinheim.

40


JOINT UNESCO-UNICEF MONITORING PROJECT

Studies and Working Documents

No. 1 Monitoring and Surveying Learning Achievements - A Status ReportVinayagum Chinapah, September 1992

No. 2 Drafts Instruments for Assessing Learning Achievements - SurveyQuestionnaires and TestsPaul Pawar, Holger Daun, Zhao Shangwu and Chen Xiaoda, November 1992

No. 3 Prototypes de questionnaires, réflexions de méthodeChristine Audouin-Leroy, Cheick-Omar Fomba et Nadera Hajji, Janvier 1993

No. 4 Instruments for Assessing Learning Achievement - Some BasicConsiderationsSusanne Schnüttgen, February 1993

Other publications

UNESCO (1993) Project Overview and the First Five Countries Application

UNESCO (1993) Manuals and Guides: Sampling, Test and Questionnaire Construction(English and French)
