Average Ignorance? Some thoughts on Multiple-Choice QuestionnairesAuthor(s): Colin JohnsonSource: Mathematics in School, Vol. 27, No. 2 (Mar., 1998), p. 5Published by: The Mathematical AssociationStable URL: http://www.jstor.org/stable/30215346 .

Accessed: 07/04/2014 11:37

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact support@jstor.org.

.

The Mathematical Association is collaborating with JSTOR to digitize, preserve and extend access toMathematics in School.

http://www.jstor.org

This content downloaded from 173.69.19.125 on Mon, 7 Apr 2014 11:37:16 AMAll use subject to JSTOR Terms and Conditions

AVERAGE IGNORANCE

Some thoughts on

multiple-choice questionnaires

by Colin Johnson

Questionnaires, multiple-choice tests (known to generations of pupils as 'multiple-guess tests') and other surveys are all around us, and are used to draw a huge variety of conclusions of varying validity. Here's something that troubles me though: how should we record 'don't know' type answers?

At first this seems unproblematic. But consider the follow- ing situation: a few days ago I was stopped by a student (at a university that I shall not name!) doing a survey about zoos. One of the questions asked something like 'when you last went to the zoo, do you remember it being "very small, small, medium sized, large, very large?" '. I wasn't really able to remember how big the zoo was at all - my answer was a 'don't know'. However, I was requested at this point to circle 'medium sized', at which I was somewhat peeved, and I attempted (ultimately unsuccessfully) to explain why I was unhappy. A rough transcript of the conversation ran some- thing like this.

'Assume you get 100 answers, of which half are people who have been to the zoo recently and can make a sensible answer to this question. Furthermore let us assume that the size of the zoo is unambiguously very large, and therefore 50 people answer very large. The other fifty people answer don't know and are therefore recorded as medium sized. At the end of all of this you are left with a mean average of large which is misrepresentative of the people who did know what they are doing.'

'Yes, but it is unfair to say that all the quizees will answer very large. They will make all sorts of answers, and therefore it'll average out okay.'

'No, that can't be right! The whole point of doing a survey is that there will be some trend in your results. If your basic assumption is that all the results will cancel each other out, then you might as well begin by saying that the result is medzium.'

'But that doesn't make any difference. We are not meas- uring what the actual size of the zoo is, we're measuring people's opinions of what they think the size of the zoo is.'

'Even so, you can't just assume that people who have no opinion about something think that something is average. If I go down the street asking everyone whether they think that the problem of the Riemann Hypothesis will be resolved this century, and most reply 'I don't know' then it's not true to say that the average person thinks that there is a middling chance that the problem will be resolved. When you carry out a survey, whether it is to find out some underlying truth (like an election poll, which attempts to find out an underlying truth of 'how people vote') or whether it is to attempt to find out what 'the average punter' thinks, then you are making an underlying assumption that you will find some trend in the results, not just that there would be a uniform distribution of results across the different options! If you really need an answer for everyone then you should make the don't knows put down a random answer.'

Let us think about this. What were the fundamental errors made here? I think that ultimately there were three misun- derstandings.

1) There was a confusion between the idea of an 'average', which is a statistic derived from data, and 'medium', which is a concept built on people's previous experience of things of a similar type. What the student meant by asking the don't knows to put down medium was that the quizee should put down the average value - which is of course impossible.

2) There is a misunderstanding about the nature of a don't know response. The difference between having the knowl- edge to make an informed decision that something is 'middling' and merely knowing nothing is important. If I am wanting to sell my antique violin/painting/camcorder then I would be inclined to believe it if ten experts told me that it was an average piece of kit which would fetch a middling price at auction. On the other hand I would be underimpressed if ten mathematicians told me that they knew nothing about the value of the thing, I wouldn't at all think that it was of middling value - it might be a van Gogh (violin!) or it might be a heap of tripe.

3) There was a confusion between the concept of'hard data' and 'opinion', which in this case were essentially the same thing. If you are attempting to gauge what people think of something, then this is the data that you are gathering, the underlying 'data' (the size of the zoo or whatever) is essentially irrelevant. Indeed there are two reasons for taking 'opinion' type data - on one hand it may be to find out genuine underlying data (for example a survey on what brand of toothpaste you prefer, so that a company can find out what proportion of the toothpaste market they corner). On the other hand it may be to find out people's actual opinion - if a company finds out that people believe that they are the biggest company in the field, when more accurate data from sales shows that they are second best, then that is likely to be seen to be a good thing (this reminds me of the old advertising slogan 'we're second biggest so we try harder'). Knowing which of these reasons a survey is being used for is important.

It is unfortunate that such misunderstandings should arise - unfortunate, but understandable, for the points are rather subtle at first. Rather more unsettling is that the misunder- standings can arise, persist and be defended, as if I was talking nonsense or being needlessly pedantic.

Fortunately the student was a student of management, not of mathematics - though perhaps it is somewhat unsettling to assume our future captains of industry failing to under- stand these things! If a survey tells an executive that people think their new product is 'fairly good' rather than 'fandabi- dozi' because the top marks have been tempered by the relentless tendency of the don't knows to drag everything towards the average, then bad management decisions could be made. Mathematics and statistics give both important skills and key understandings for decision-makers in all walks of life, not just an elaborated IQ test.

What can we do to prevent this kind of misunderstanding arising? This needs to begin in the schools - we need to develop a questioning, critical attitude towards mathematical concepts such as 'average'. The examination in the classroom of case studies such as the conversation above offers one possible route towards a deeper understanding of these fun- damental concepts. I would welcome thoughts of other such approaches. a]

Author Colin Johnson, Department of Computer Science, University of Exeter, The Old Library, Prince of Wales Road, Exeter, EX4 4PT.

Mathematics in School, March 1998 5

This content downloaded from 173.69.19.125 on Mon, 7 Apr 2014 11:37:16 AMAll use subject to JSTOR Terms and Conditions