Student Learning Study – Key Findings...Student Learning Study – Key Findings Vyjayanthi Sankar Vice-President – Large Scale Assessments Educational Initiatives Pvt. Ltd. [email protected]

Student Learning Study – Key Findings

Vyjayanthi Sankar Vice-President – Large Scale Assessmentsg

Educational Initiatives Pvt. [email protected]

New Delhi, May 24th, 2010

"The search for truth is in one way hard and in another way easy, for it is evident that no one can master it fully or miss it wholly. But each adds a little to our knowledge and from all the facts assembled there arises

certain grandeur.” --- Aristotle

Need for Quality Education

Learning outcomes

Nation is dependent on the knowledge capital of its citizens for growth

Learning outcomesGDP increases significantly with moderately strong knowledge improvement

Years of schooling

Q tit Q litQuantity vs Quality

Being Ready for 21st Century

According to an article in Time Magazine, “in order to competein the 21st Century, students need to be taught a curriculumthat balances core knowledge such as math science andthat balances core knowledge, such as math, science andreading, with what educators call “portable skills,” such ascritical thinking, making connections between ideas and knowinghow to continue learning”.

Speed of Innovations in 21st century –to be more intense

how to continue learning .

Advancement depend more and more on ability to innovate,patent, etc.

N d t hift f i iti f K l d t i iti fNeed to shift from acquisition of Knowledge to acquisition ofProcess. (logical reasoning, analytical thinking, creativity,learning to learn to support independent lifelong learning)

Need for the Study

We need studies that tells us

What are the levels of student achievement?

Are children really learning and understanding what isy g gtaught to them?

How much are they retaining? Is learning happening at asuperficial level?

What do children at different levels of attainment know andbl t d ?are able to do?

Are children of some states do better than others, if sohow?how?

About the Study

Class 4

Student Learning Study - Objectives

To provide independent 3rd party benchmarking of studentlearning for use by policy makers and researchers both atlearning for use by policy makers and researchers both atthe Central and the State levels.

To provide insights into comparative performances ofp g p pdifferent states and to create cross-learning and remedialopportunities.

To establish student learning in terms of knowledge andskills and feedback on the learning gaps, common errors andmisconceptions.

To provide pointers for further research.

To diagnose the learning issues at the systemic levelTo diagnose the learning issues at the systemic level.

Basic Facts about the Study

Study design entirely conceived, conceptualised andexecuted by Educational Initiatives

Conducted with the support of Google.org, USA

Tests conducted with permission of state governmentsTests conducted with permission of state governments

Tests for Classes 4, 6 and 8

Subjects: Language and Maths

Test Paper Versions: in 13 Languagesp g g

Tests between December 2008 – July 2009

Experts involved in different stages. For example, ISI inSampling, CIIL experts in vernacular tool development

How is the study different from other studies?Class-wise tests Class wise testsSingle Agency Facilitated and carried out by Educational InitiativesConducted In classroomsCoverage Classes 4,6,8 in Urban and Rural schoolsTesting Tools *Sophisticated , scientifically designed, full length tools

*Additional oral reading testBackgroundFactors

Separate questionnaires for information about the student,school, teacher and head teacher

Analysis *Focus on moving beyond the average scores to provided f db k l ddiagnostic feedback on learning gaps and misconceptions

*Comparison on anchor questions from internationalTIMSS and PIRLS

*Identification of factors associated with learningScale Anchoring Scale Anchoring using modern Item Response Theory (IRT)

to release benchmarks on what students know and do atdifferent ability levelsdifferent ability levels

Video Study Student interviews in 3 states to analyse studentmisconceptions and common errors

Coverage

18 states and 1 UnionTerritory

WB, HP & UP did notprovide permissions for thestudystudy

Delhi participated only inClass 4Class 4

2399 Urban and RuralGovernment schoolscovered

1.6 Lac students sampled;about 1 Lac actually tookthe test (Attendance ratewas 66%)

Sampling

Enrolment data received from DISE and supplemented by listsfrom states, for eg., Chandigarh schools

States with more than 1% of the overall population selected

Districts ranked using Human Development Indicators/ LiteracyRate and divided in 4 or 2 groups as per the rank – high tail,above median, below median, low tail.

4 to 2 districts selected in each state one from each group4 to 2 districts selected in each state – one from each group

Two stage Stratified Cluster sampling design implemented tostratify schools and selection of classroomsst at y sc oo s a d se ect o o c ass oo s

Schools selected as per ‘probability proportional to size’ (PPS)and intact classrooms tested

Sample size accounted for ‘design effect’ (95% confidence level,+ 5.0 percent precision)

Test Design

‘Written’ and ‘Group Oral’ components

Multiple question formats (not only MCQ or traditional)Multiple question formats (not only MCQ or traditional)

Common Anchor questions from International tests –TIMSS, PIRLS

Class 4 and 6 questions drawn extensively from earlier Municipal school benchmarking study (in which 35000 students across 30 towns in 5 states were tested)

Question TypesPrevious and earlier class competencies (75-25)Straightforward and non-straightforward (60-40)R lRepeats across classes

Scoring Process captures common student errorspminimises evaluator errorqualitative analysis of a few free response questions in a sub-sample

Sample Questions

Class 6 Class 6 MathsLanguage Class 6 Maths

Questions check for deeper understanding of concepts

In which of these numbers below does 3 have the greatest place value?

Class 6

Which words below best describe the

Language Class 6 Maths

A. 136B. 239C. 301

wolf?A. greedy and cleverB. innocent and hungryC foolish and greedy C. 301

D. 743C. foolish and greedyD. clever and hungry

Sample Questions

Cl 6 M th

Questions check for learning that is straightforward / text-bookish

Class 4 Class 6 Maths

Write the answer.

Maths Class 6 Maths

What is the perimeter of this shape?

39- 6

________

shape?

________ _________ cm.

Sample Questions

Class 6Class 6 Maths

Questions check for application of concepts

Class 6 Maths

Look at this calendar.

What day will 1st April be?

_________

Sample Questions

Class 6Class 8 Language

Questions check for reasoning ability

Class 8 Language

Do you think the lion liked the hare? What happens in the story that shows this?y

______________________________________________________

______________________________________________________

Sample Questions

Class 8 Class 6 MathsLanguage Class 6 Language

Questions from ‘unseen’ passages and authentic material

The pickle should not be eaten after the month of

Class 8

What is the main message of this story?

Language Class 6 Language

________________. (write the month and the year)

A. Run away from trouble.B. Check the facts before panicking.C. Even lions that seem kind cannotC. Even lions that seem kind cannot be trusted.D. Hares are fast animals

Sample Questions

Class 6 MathsCl 4

Questions check for holistic language learning and basic writing ability

Write 3 sentences about this picture


Complete the passage using suitable words of your own.

Language Class 4 Language

this picture. suitable words of your own.

Ganga makes and sells garlands. Shebuys the flowers from the _______

d h k h l dand she makes them into garlands.She uses flowers of different_________. But she usually makesgarlands of small white flowersgbecause they ____the most. Shesells ______ to many people. Shesells all the garlands she makes forth dthe day.

Sample Questions


Maths Class 4 Language

International Benchmarking questions

How does the story show you what the mice

Class 8

One centimetre on the map represents 8 kilometres on the

Maths Class 4 Language

thought was happening?

A. by telling you what Labon

land. About how far apart are Town A and Town C on the land?

A. by telling you what Labon thought of the miceB. by describing where the mice livedC b t lli h t th4 C. by telling you what the mice said to one anotherD. by describing what the mice were like

A. 4 kmB. 16 kmC. 35 kmD. 50 kmD. 50 km

Oral Reading Test

For Class 4 only

Ad i i d b l f d i hAdministered one-on-one to sub sample of students in each classroom

A total of 8 questions in the testA total of 8 questions in the test

3 questions on reading words of graded difficulty

2 questions on reading sentences

1 short passage p g

1 question retrieving explicitly stated information

1 question on comprehending implicit information

The Test Development StagesStages

Phase1: Information Defining the Focus Group to be TestedDetailed Study of Textbook/Syllabi

Generation based on apriori hypothesis

Study of Existing Norms & Standards Comparative Study of Other Curricula & Tests Item Generation and Development

Phase II: Item Ordering, Paper Development, Pretesting for Face

Pre-test Instrument Development Pre-testing Test Design & Paper Revisiong

Validity and Cognitive Debriefing

Pilot – Field Tests

Test Design & Paper Revision(components, difficulty level, Testable competencies-Level & preparation , Test length)

Pilot Field Tests Statistical and Qualitative Analysis,Finetuning Papers, Scorecards, etc Translations & Validations

Phase III: Final Instrument Development & Psychometric Validation

Linguistic Harmonisation – Iterative Final Instrument

y

Test Administration

Evaluators and Team Leaders recruited through a rigorous process of selection

Master Trainer Workshops and Evaluator Workshops for training - provides for hands on experience in test administration and coding Field trials and mock testingadministration and coding - Field trials and mock testing

Quality control through detailed checklists, manuals and field auditsaudits

Documentation of non compliance in test administration or answer coding - for consideration during data analysisg g y

Use of OMR technology for data entry

Specialised software programs for identification of data errors

Main Findings

Low Levels of Learning

In lower classes – students are able to comparatively handlequestions that are very ‘straightforward’ and NOT when theyare slightly atypicalare slightly atypical.

MathsClass 4

What time is the clock

MathsClass 4

Write the answer.showing?

70- 43

____________

LanguageClass 6 Class 6

71.4%________

________ 49.4%

Language

Class 4Kheer is __________ to taste. A. bitter B

Fill the blank with an appropriate word.

Q i i h b k?B. sweet C. spicy D. sour 89.6%

Question: ………is the book? Answer: It is on the table.

20.9%


In higher classes – students are falling behind in all learning,even straightforward, procedural questions.


The ratio of boys to girls in a class is 2:3. If there are 12 girls, what is the number of boys?

16.0%


CClass 6 Maths

Cl 6

Students are probably coping through learning outside the class

Class 6 Maths Class 6 Maths

Write the answer. Ravi bought 2½ kg rice

2 + 1 = ________21

21

from one shop and1½ kg rice from anothershop and put it in his bag.Wh i h l i hWhat is the total weightof rice he bought?

_________ Kg

34.6%20.5%

Main Findings – Type of Learning

CClass 6 Maths

Cl 4

Whatever little learning that is taking place is not ‘Learning with Understanding’

Class 4 Maths Class 4 Maths

Procedural Q i

Conceptual Question

Write the answer.

43

Question

Fill in the appropriatenumber in the box.

43 × 2

____________3 × = 3 + 3 + 3 +3

____________

30.4%67.1%

Type of Learning

Class 6

Whatever little learning that is taking place is not ‘Learning with Understanding’

Class 8 Language

Passage Excerpt:An old man who lived in the city had to put up with the nuisance of having boys play cricket in the streets outside hishouse in the streetlights, at night.One evening, when the boys were particularly noisy he could not sleep. So he went out to talk to them. He explainedthat he was very happy whenever he could see or hear boys playing his favourite game, cricket. He said he would givethem 25 rupees each week to play in the street at night.The boys were thrilled.They were being paid to do something they enjoyed!At the end of the first week ….he did so.The second week …..away with only 15 rupees.The third week ……gave them only 10 rupees.The …..fourth week …..would give them 5 rupees each week without fail.……….The boys then stormed away and never played on the street again.

How much money did he promise Wh t did th ld ll t f thHow much money did he promise to give the boys?

A. 25 rupees every week

What did the old man really want from theboys?A. He wanted them to play cricket on all thedays of the week.B. He wanted them to practice more and thusimprove their game

74.4% 45.7%

B. 15 rupees every weekC. 10 rupees every weekD. 5 rupees every week

improve their game.C. He wanted them to stop playing cricket nearhis house at night.D. He wanted them to start earning while doingsomething they liked.

Learning across ClassesLearning gains seen across classes is slightly incremental and not a large jump

Language

Learning across Classes

Class 6

Misconceptions persist in higher classes

Class 4, 6,8 Maths

Main Findings – Writing

Students writing does not go beyond the most basic,tried and tested formulaic sentences they probablytrained for while in their lower classestrained for while in their lower classes

Their writing shows that they are learning languagemore as a subject and less as a means of naturalmore as a subject and less as a means of naturalcommunication.

The writing has a number of errors in spelling andg p ggrammar, and punctuation marks are conspicuous bytheir absence.

Oral Reading TestIn all states tested, fewer students were found to comprehend what they read.

Language

Comparative Performance of StatesThere are significant state-wise differences in student performance.

Language

The union territory of Chandigarh, although tested has not been included in this comparison.Delhi is also not included in this comparison as only class 4 was tested here.

Comparative Performance of StatesExtent of students scoring zero and the overall performance of a state showed differences.

Language

The union territory of Chandigarh, although tested has not been included in this comparison.Delhi is also not included in this comparison as only class 4 was tested here.

Comparative Performance of States

Language

If a state relatively did well in a classdid well in a class and subject, it was likely to do relatively well in yother classes and subjects too.

Comparative Performance of Boys and Girls

The comparative performance of boys and girls is similarto many international studies – boys seem to do better inMaths although the SLS study reveals that it is aLanguageMaths, although the SLS study reveals that it is ameaningful difference/a difference that matters in class 8.

Comparison with International Performance

The levels of learning is much

Languagelower than the international average on

i dquestions used from TIMSS and PIRLS.

SAMMPLE

Language

BENCHMARKK

ITTEM

Knowledge and Ability Benchmarks – Scale Anchoring

Language

Knowledge and Ability Benchmark DescriptionsAdvanced Benchmark (students reaching 90th percentile)

90th Percentile IRT Scaled Score: 626

Students understand the concept of place value and know that a 3 digit number can be represented in expanded notationas a sum of the number of hundreds, tens and ones in it. Students understand the concept of multiplication as repeatedaddition and are able to equate multiplication of 2 numbers as the number of times another number is added. Studentsknow how to form groups of equal numbers out of a given set of an object. Students understand half represented as afraction and know that it as one out of 2 equal parts and are able to identify the correctly shaded figure based on this.

Languageq p y y g

Students can visually identify simple geometrical shapes such as triangles, in a tilted orientation based on their understandingthat it is a shape enclosed by 3 straight lines. Students are able to perform conversions between simple units ofmeasurement such as millimetres and metres. Students are able to evaluate and choose the correct mathematical operationto be applied to solve a word problem which requires multiplication of 2 single digit numbers.

L B h k ( t d t hi 25th p til )Low Benchmark (students reaching 25th percentile) 25th Percentile IRT Scaled Score: 439

Students write the numeral form of 2 digit numbers when their number names are given. Students know2 digit numbers less than 20 and can identify a number that is missing from a sequence of consecutive2 digit numbers less than 20 and can identify a number that is missing from a sequence of consecutivenumbers. Students complete the missing numbers in a descending sequence of 2 digit numbers less than20. Students know the arithmetic operations of addition and subtraction and are able to carry them outfor 1 digit numbers, placed vertically. Students do addition of two 2 digit numbers, placed vertically andinvolving carry over. Students do subtraction of 1 digit numbers from 2 digit numbers, placed verticallyg y g g , p yand not involving borrowing. Students know the arithmetic operation of multiplication and are able toperform the same for 1 digit numbers, placed horizontally. Students identify similarities and differencesvisually, between different 2D shapes. Students understand fractional quantities such as half written in aword form as one out of 2 parts and apply them practically in their daily context. Students read time froma traditional analog clock. Students are familiar with the traditional currency denominations and canperform simple addition of the same. Students can read the price tags of objects whose values arerepresented by 3 digit numbers, to find the one which costs the least. Students understand wordproblems based on counting the number of objects present in loose and in bundles of ten.

How is this information useful?

LanguageAt the policy level, benchmarks determine the focusarea for teacher training. For eg., while MP has tofocus their teacher training for Maths 4 teachers onfocus their teacher training for Maths 4 teachers onthe knowledge and ability described in low benchmark,TN has to focus on the advanced benchmark


21

Language


The teacherLanguage The teachercan use thedifferentbenchmarkitems in theclass tests todeterminewhetherstudents ofher class arebl hable to reach

them

Background FactorsMeans of students scores in Language and Maths

crossed with variables in student questionnaire

Language

Background FactorsMeans of students scores in Language and Maths

crossed with variables in student, school questionnaire

Language

Background FactorsAnalysis of Background factors showed that when studentsperceive themselves as being good at studies and think ofschool as a place of fun and learning they show goodLanguageschool as a place of fun and learning, they show goodperformance.

The analysis also reveals the importance of inculcatingy p greading habits in student, for spending 30 minutes each dayto read material other than textbooks brings about higherachievement.

Teacher training is important especially for mathematicsteachers for better achievement of students in the subject.

Unexpectedly, analysis shows that students whose teachershave an academic degree developed lower scores in bothlanguage and mathematics than those who did not havelanguage and mathematics than those who did not havethese

Limitations of the Study

The issue of student seriousness towards a low-stakes testLanguage

Testing limited to language and mathematics

Strengths and Weaknesses of the System

Strengths

Fairly qualified teachers (out of 2399 schools tested, 75% ofh h d i d d 4 2% dLanguageteachers have an academic degree or more, and 4.2% reported not

having any teacher training).

Education is valued in the societyLearning in mother tongue

Ability of the state to handle large programmes

Weaknesses

Focus on rote learning

Students lagging much behind in higher classesStudents lagging much behind in higher classes

Focus on ‘inputs’ rather than ‘outcomes’

No empirical research to drive system shift

Too much diversity/tiers of schools

Low teacher accountability and motivation

Recommendations

Orienting policies to focus on learning outcomes

Making low stakes diagnostic assessment a regular feature ofLanguagethe state programs

Extensively use benchmark data from the study to buildreformreform

Building a Student Progress Tracking system

Large scale awareness campaigns to refine attitudes towardsg p glearning – a movement against rote learning and for learningwith understanding

Widely disseminating the findings of this report amongWidely disseminating the findings of this report amongteachers and others

Providing effective teacher support based on the feedbackf th tfrom the assessments

Thank YouThank You

“If you learn how to look at data in the right way, you can explain riddles that otherwise might have seemed impossible. Because there is nothing like the sheer power of numbers to scrub away layers of confusion and contradiction.”

-S.D. Levitt

Background QuestionnairesI C V i bl D i i Q i iItem Context Variable Description Questionnaire

School Type of school School

Multi-grade classroom School

Infrastructure School

Material and equipment School

Hours per year Documents

Mid-day meals program School

Head-teacher Gender Head-teacherAge Head-teacherAcademic qualification Head-teacher

Teacher training qualification Head-teacher

Years of experience Head-teacher

Type of service Head-teacher

Teacher Gender Teacher

Age TeacherAcademic qualification TeacherTeacher training qualification TeacherYears of teaching experience TeacherType of service TeacherPerception on discipline Teacher

Student Gender StudentAge StudentSocioeconomic background StudentParental occupation Student

Tuitions Student

St d t P ti P ti b t h l St d tStudent Perception Perception about school Student

Liking for the subject Student

Use of school library Student

Student’s reading habits StudentPerception about their own academic performance Student