Assessment of skills and competences in science

Wynne Harlen

OECD Skills Strategy“In the context of the OECD Skills Strategy, the concepts

of ‘skill’ and ‘competence’ are used interchangeably. By skill (or competence) we mean: the bundle of knowledge, attributes and capacities that enables an individual to successfully and consistently perform an activity or task, whether broadly or narrowly conceived, and can be built upon and extended through learning.”

Towards an OECD Skills Strategy (OECD 2011 p7 foot note)

AgendaStarting point: aims of science units and courses (Access and

Higher)Inquiry as a means to developing scientific literacyResearch into development of understanding through inquiryChallenges to assessing understanding and skills (particularly

context sampling error )Examples of questions assessing skills and understanding in

scienceAlternatives to external papersPros and cons of using teachers’ assessment.

Unit and course outcomesUnit assessment“Learners who complete this Unit will be able to:

apply skills of scientific inquiry and investigation in the context of the unit (eg physics of dynamics and astrophysics)

demonstrate knowledge and understanding related to the content of the units (eg physics of dynamics and astrophysics) “

Scientific literacy as an aim“The Course will therefore encourage learners to become

scientifically literate citizens, while developing their literacy and numeracy skills. It will also develop learners’ investigative and experimental skills in a (physics) context. Learners will recognise the impact (physics) makes on their lives, the environment and society. Through this Course, learners can develop relevant skills for learning, for use in everyday life and across all sectors of employment.”

A note on scientific literacy (SL)Scientific literacy as a metaphor in which the traditional

sense of literacy assumes another meaning (Bybee).SL refers to being well-educated and well-informed in

science. It does NOT mean being able to read and write about science or being able to understand science vocabulary.

Development of SL: scientific illiteracy --> nominal SL --> functional SL --> conceptual and procedural SL --> multidimensional SL

Through scientific inquiry students’ advance their understanding of scientific concepts and the nature of science, that is, procedural literacy.

The role of inquiry in science education

Through inquiry students progressively developing key scientific ideas through learning how to investigate and build their knowledge and understanding of the world around. They use skills employed by scientists such as raising questions, collecting data, reasoning and reviewing evidence in the light of what is already known, drawing conclusions and discussing results. (based on IAP 2011)

Is it necessary to assess inquiry skills? Yes, if inquiry skills and competencies are seen as key

outcomes in their own right, not just a means to developing understanding (of science and about science)

Skills needed for continued learning –Students cannot learning in school everything they will need to

know in adult life. What they must acquire is the prerequisites for successful learning in future life….Students must become able to organise and regulate their own learning, to learn independently and in groups and to overcome difficulties in the learning process. This requires them to be aware of their own thinking and learning strategies and methods. (OECD 1999)

Existing idea Possible explanation

Prediction

Plan and conduct investigation

Interpret data

New experience/question

Conclusion

Alternativeideas

Bigger idea

How does this development of understanding happen?

Does inquiry ‘work’?Do inquiry-based activities lead to understanding more

than do more direct teaching approaches?Inquiry synthesis project

Minner, D.D., Levy, A.J., & Century, J. (2010). Inquiry-based science instruction – what is it and does it matter? Results from a research synthesis years 1984-2002.

Journal of Research in Science Teaching, 47 (4), 474-496

Inquiry Synthesis ProjectSynthesis of 138 studies

• 30 experimental; 35 quasi-experimental; 73 non-experimentalMost (105) conducted in the United StatesAmount of instruction delivered in the interventions varied

widely (mean ~ 12 lessons)Most (56%) did not report if the outcome measurement

instruments were new or existing• Generally low rigour regarding reliability and validity of the

measures used

Inquiry Synthesis Project findingsThe majority of studies (51%, n=71) showed positive impacts of some

level of inquiry science instruction on student content learning and retention

The extent of ‘inquiry saturation’, or specific elements of inquiry, or specific components of instruction was not statistically significantly associated with better learning outcomes

Active thinking was a significant predictor of learning outcomesDrawing and thinking about conclusions from data was a marginally

significant predictor of learning outcomes5 studies (of the 6 that were reviewed) of the effect of hands-on

activities on student learning showed a statistically significant difference

6 studies (of the 9 that were reviewed) of the effect of giving students greater responsibility showed a statistically significant difference

Course assessment

“Learners will draw on, extend and apply the skills they have learned during the Course.

This will be assessed withina question paper (external paper)and a case study (stimulus-based assessment of ability to

analyse and draw conclusions)requiring demonstration of the breadth of knowledge,

skills and understanding acquired from across the Units and of how they can be applied in unfamiliar contexts and/or integrated ways.”

Challenges for assessment

To show that the skills, knowledge and understanding can be used in ‘unfamiliar contexts’

The need for authentic situations, real contextsComplexity in presenting the contextBias of familiarity with the contextPreference for ‘unfamiliar contexts’ in order to assessment

application of understanding and capabilitiesContext sampling errorImpact on reliability and validity

Sources of error in assessment of understanding Requires application of knowledge, in a contextHow to choose – what are ‘unfamiliar contexts’?Lack of knowledge or lack of ability to apply?Context sampling error (Wiliam 2001).

The context effect in assessment of inquiry skills(Brown et al 1996)

‘Little evidence of the generalisability of skills assessment across science subjects’

Need for continued assessment of skills within subject domainsHigh context sampling error when few contexts can be used.Possible solutions:

Find contexts likely to be unfamiliar to all studentsIncrease contexts by assessing smaller, specific aspects of inquiryUse the range of contexts encountered in the course for

moderated teacher’s assessment (e.g. portfolio).

Important differences between assessing populations and assessing individualsPISA and national monitoring programmes

Can minimise the context sampling error by using a bank of items and random allocation of items to students

But, restricted in the form of the item on paper or on screenFor population, high reliability (accuracy) but lower validity

Assessing individualsUsing examinations and tests, same items of all, limited range of

contextsLarge context sample error; low validityUsing performance in regular and special inquiry-based tasks raises

validityHigher reliability tends to lower validity.

What Human activities contribute to climate change?

Read the following information and answer the questions which follow.

Cooling Relative Importance Heating

Carbon dioxide

Methane

Particles

Particle effects on clouds

known effectpossible effect

The burning of coal, oil and natural gas, as well as deforestation and various agricultural and industrial practices, are altering the composition of the atmosphere and contributing to climate change. These human activities have led to increased concentrations of particles and greenhouse gases in the atmosphere. The relative importance of the main contributors to temperature change is shown in the figure

1. Use the information in Figure 1 to support the view that priority should be given to reducing the emission of carbon dioxide from the human activities mentioned.

2. Use the information in Figure 1 to support the view that priority should not be given to reducing the emission of carbon dioxide from the human activities mentioned.

Bars extending to the right of the centre line indicate a heating effect. Bars extending to the left of the centre line indicate a cooling effect. The relative effect of ‘Particles’ and ‘Particle effects on clouds’ are quite uncertain: in each case the possible effect is somewhere in the range shown by the light grey bar. Figure 1 shows that increased concentrations of carbon dioxide and methane have a heating effect. Increased concentrations of particles have a cooling effect in two ways, labelled ‘Particles’ and ‘Particle effects on clouds’.

Questions

Early immunisationAs early as the 11th century , Chinese doctors were manipulating the

immune system. By blowing pulverised scabs from a smallpox victim into their patients’ nostrils. They could often induce a mild case of the disease that prevented a more severe onslaught later on. In the 1770s people rubbed their skins with dried scabs to protect themselves from the disease. These primitive practices were introduced into England and the American colonies. In 1771 and 1772, during a smallpox epidemic, a Boston doctor named Zabdiel Boylston tested an ideas that he had. He scratched the skin on his six year old son and 285 other people and rubbed pus from small pox scabs into the wound. All but six of his patients survived.

Questions:1. What idea might Zabdiel Boylston have been testing?

2. Give two other pieces of information that you would need to decide how successful Boylston’s approach was

Chocolate A newspaper article (Daily Mail on March 30, 1998) recounted

the story of a 22 year old student, named Jessica, who has a ‘chocolate diet’. She claims to remain healthy, and at a steady weight of 50Kg, whilst eating 90 bars of chocolate a week and cutting out all other food, apart from one ‘proper meal’ every five days. A nutrition expert commented: “I am surprised someone can live with a diet like this. Fats give her energy to live but she is not getting a balanced diet. There are some minerals and nutrients in chocolate, but she is not getting enough vitamins. She could encounter serious health problems in later life.”

Imaginary animal

Threats to validityCommunicating an unfamiliar context makes demands on

reading, interpretation of representationsThe extent to which the assessment specific skills equates

to assessment of capability in inquiry and investigationsOther features of context (beyond familiarity) may affect

engagementWhat was once a ‘novel’ context can become familiar

once usedScience fiction a questionable context for assessing

understanding and skills in science.

Some advantages of assessment by teachersPotential for the full range of goals to be included as

teachers collect evidence as part of their normal work with students

Can relieve the pressure, on students and teachers, of terminal tests and examinations

Teachers can use information about students formatively as well as summatively

Can release resources (time and other costs) for alternative use

Some disadvantages

Teachers’ judgements often perceived as being unreliable Increase in work load for teachersCan lead to the same distortion of teaching as testing if the

results are used for high stakes accountability..

Dimensions of approaches to assessment by teachers

Greater specification of tasks

Assessment criteria more detailed

12

43

A portfolio system (eg Queensland)

School-basedConstructed over all units in the courseSelected and assessed by reference to strict criteriaFull range of evidence – projects, assignments,

observations of practical performance, field workMandatory criteria Up-dated to include most recent relevant evidenceFive levels of achievementModerated.

A formative functionAssessment of any kind should ultimately improve

learning (Harlen, 2010)The collection of data during the year informs the

teacher of the areas of more and less progress of students (formative use of summative information)

The practice of assessment for learning can inform the judgement of achievement at a particular time (summative use of formative information).

ReferencesBrown, CR., Moor J., Silkstone, B.E., and Botton, C. (1996) The construct validity and context

dependency of teacher assessment of practical skills in some pre-university level science examinations, Assessment in Education, 3 (3) 377-391.

Bybee, R., Fensham, P and Laurie, R (2009) Scientific literacy and contexts in PISA 2006 science. Journal of Research in Science Teaching 46 (8) 862-864.

Harlen, W. (2001) The assessment of scientific literacy in the OECD/PISA project. Studies in Science Education, Vol 36, 79-104

Harlen, W. (2005) Trusting teachers’ judgment: research evidence of the reliability and validity of teachers’ assessment used for summative purposes. Research Papers in Education 20 (3) 245-270

Harlen, W. (2010) What is quality teacher assessment? In Gardner et al Developing Teacher Assessment. Maidenhead: Open University McGraw-Hill

IAP (InterAcademies Panel) (2011) Taking IBSE into Secondary Education. Report on a conference held in York, October 2010. London: Welcome Trust.

Minner, D.D., Levy, A.J., & Century, J. (2010). Inquiry-based science instruction – what is it and does it matter? Results from a research synthesis years 1984-2002. Journal of Research in Science Teaching, 47 (4), 474-496

Wiliam, D. (2001) Reliability, validity and all that jazz, Education 3-13 (3) 17-21