We would like to thank the officials of Education ...edumis.hp.gov.in/Application/uploadDocuments/... · A team of EI assessment experts conducted training for Principals of District

© Educational Initiatives Pvt. Ltd.

1

We would like to thank the officials of Education Department and Sarva Shiksha Abhiyan, Himachal Pradesh. The successful completion of this assessment is attributed to help from many individuals in the department of Himachal Government.

This study would not have been possible without support from Michael and Susan Dell Foundation (MSDF) and Learning Links Foundation (LLF). We are highly indebted to them for their guidance and constant support for successful roll out of State wide assessments.

Special gratitude is due to all the DIET Principals and officers, Research team-SSA and LLF team at Himachal for their kind co-operation and the great support. We were able to carry out the assessments smoothly because of the great support provided by the all the Government school teachers who carried out the test administration.

Last but not the least; we are also thankful to the students, and principals of the schools of Himachal that participated in the Assessments.

We sincerely hope that this illustrative study will be useful to the policy makers and schools.

MR. SRIDHAR RAJAGOPALAN MR. VISHNUTEERTH AGNIHOTRI

(Managing Director) (Vice President - Assessments) Educational Initiatives Pvt. Ltd. Educational Initiatives Pvt. Ltd.

ACKNOWLEDGEMENT


2

Himachal State Education Department

Sarva Shiksha Abhiyan, Himachal Pradesh

Michael and Susan Dell Foundation

Learning Links Foundation

Educational Initiatives Pvt. Ltd.

Mr. Sridhar Rajagopalan, Managing Director Mr. Vishnuteerth Agnihotri, Vice-President, Assessments Ms. Vyjayanthi Sankar, Vice-President, Strategic Relationships Mr. Venkatesh Reddy, Manager Communications Ms. Neha Miglani, Research Associate Ms. Archana Dwivedi, Research Fellow Ms. Bharti Porwal, Educational Specialist Ms. Swati Jha, Education Specialist Mr. Nishchal Shukla, Educational Fellow Ms. Sidhya Balakrishnan, Jr. Research Associate Ms. Kaveri Malik, Programme Implementation Manager Ms. N.V.N.S.S. Durga, Graphic Designer Mr. Rahul Mathur, Software Development Manager Mr. Jaikishan Keswani, Jr. Software Engineer Ms. Jayashree T.S., Educational Specialist Mr. Devpal Shah, Manager- Design and Printing Mr. Ketan Beherawala, Asst Manager - Accounts Mr. Drupad Shah, Manager - Logistics

TEAM


3

EXECUTIVE SUMMARY 4

1. DESIGN AND IMPLEMENTATION OF THE STUDY 6

1.1 Need For The Study 6 1.2 About the Study 7 1.3 Salient Features Of The Study 8 1.4 Question Paper Design 9

2. MAIN FINDINGS 15 2.1 Performance Across Subjects and Classes 15 2.2 Comparative Performance Against Indian Private Schools 15 2.3 Learning with Understanding - Performance on Higher Order Questions 16 2.4 Performance On Practical Competencies 18 2.5 Performance Across Districts 19 2.6 Performance Of Boys And Girls 21

3. INSIGHTS ON SPECIFIC LEARNING AREAS 22

3.1. Misconceptions And Common Errors 22 3.2. Performance In Different Skills 38

4. FIELD AUDITS 42

5. A PEEK INTO THE DISTRICT REPORTS 46 5.1 A Bird’s Eye View Of The District’s Performance Across Classes And Subjects 46 5.2 Score Distribution Of Students Across Subjects, Class 47 5.3 Comparison With Other Districts (Composite Index) 48 5.4 Performance On Different Skills 49 5.5 Question-wise Performance Across Classes And Subjects 50 5.6 Subject Level Recommendations 51

6. APPENDIX 52

Appendix A: Comparison With Asset 52 Appendix B: Difference In Performance In Straight Forward (SF) And Higher Order (HO) Questions 52

Appendix C: District-wise Performance 53 Appendix D: District-wise Composite Performance 54

CONTENT


4

Why this Study?

Understanding the importance of a comprehensive educational policy, the state of Himachal Pradesh conducted an assessment of learning levels with Educational Initiatives, in partnership with the Michael and Susan Dell Foundation and support from the Learning Links Foundation. The Learning Assessment was conducted in June 2013 in an attempt to undertake a scientific and systemic study to detect state learning levels. . It seeks to provide independent third-party assessment of student learning to be used by policymakers and researchers and establish student learning in terms of knowledge and skills (or competencies) and provide feedback on the learning gaps, common errors and misconceptions. This report will argue that assessment must promote good education rather than undermining it.

Design of the study

Coverage: The study was aimed to cover all the 12 districts in Himachal Pradesh. Around 1000 students per class from each district were sampled and tested for classes 3, 5 and 7. Diagnostic assessment was conducted with Hindi as the medium of instruction. The study included a detailed scientific process of test development supported by curriculum checking; development of test items based on the previous studies conducted by EI, fine-tuning and assembling of the final papers; detailed test adaptation, translation, validation of papers; master trainer workshops, field audit, data entry, and diagnostic data analysis prior to the release of the report.

Test Format: The learning assessment was conducted for two subjects-Language (Hindi) and Mathematics for classes 3, 5 and 7. These test papers were designed for a wider audience of MSDF grantees. For each class and subject, one set of papers was made and adapted into Hindi language. The question papers included straightforward questions as well as higher order questions detecting deeper understanding and application of learnt concepts. The test papers were completely written but the question paper was read aloud once in Class 3

Students of class 3 filled in their answers on the question booklets itself. At class 5 and 7 level, the students directly marked their answers in the given OMR sheets. However, they answered the free response questions in the question booklets. After the test, the students’ responses (OMRs as well as question booklets) were sent back to EI where the class 3 complete question responses and class 5 and 7 free response questions were coded on OMR sheets by the evaluators at EI.

Test Conduction and Analysis: All schools got the same paper to detect learning levels and to undertake a comparative analysis of student performance. Further, in order to benchmark achievement of Himachal Pradesh students with those of Indian private schools (not low-cost schools), sufficient number of test items were used from ASSET1

1 ASSET is a diagnostic test from Educational Initiatives, which is administered every year to more than four lakh students studying in Indian private schools (not low-cost). Details can be obtained through www.ei-india.com.

.

A team of EI assessment experts conducted training for Principals of District Institute of Education & Training (DIET) as well as some District Primary Education Officers (DPEOs), District Education Officers (DEOs) to ensure standardized test administration. Test conduction was managed by the State of Himachal Pradesh with the help of LLF. EI teams also carried out field audits during the day of the test in over 3 schools spread across 5 districts.

EXECUTIVE SUMMARY


5

Based on the student performance on these tests, different types of analyses were carried out using advanced statistical methods to extract patterns in performances and to understand differences in learning levels across different groups and patterns in learning. The findings from this analysis have been presented in this report.

MAIN FINDINGS

Based on the analysis of student performance at the school, district as well as the state level, several key findings have been presented below and detailed in the sections that follow.

1. The state average performance ranges from 48.5% to 59.5% in Hindi and 38.1% to 59.7% in Maths across the three tested classes. It was observed that the performance was more homogenous within higher classes, (indicated by lower standard deviation).

2. Students in the State perform lower than the Indian Private School Average (assessed through ASSET), typically by 5-13 percentage points. Further, this gap between the state and ASSET performance increases in the higher classes.

3. Lower performance of students was observed in higher order questions involving deeper understanding of concepts. This indicated that learning with understanding is weak, as the students find it difficult to apply learnt concepts in unfamiliar question formats.

4. The students were observed to be the weakest in questions testing practical competencies such as visual interpretation, map reading, observation and estimation. Practical competencies are essential for the students’ all round development and this is a matter of concern.

5. A comparative analysis of the performance of all the districts revealed that Hamirpur outperformed all other districts, performing significantly better than the state average. At the same time, the districts of Chamba, Bilaspur and Sirmaur observed lower performances, significantly below the state average.

6. Students seem to have acquired several misconceptions during the process of learning. These were observed amongst the students across competencies and subjects. These misconceptions can arise at any time, and should be tackled at the earliest through strategic pedagogical improvements.

7. The student performance was weak in particular skills of Hindi and Maths. These include intermediate skills of Fractions, Surface Area and Volume and Integers in Maths and reading comprehension, particularly in inference based questions and passages based on authentic material in Language.

These findings have been comprehensively discussed in chapters 2 and 3 of this report. The report will present these findings with relevant statistical methods, representation and sample questions.


6

1.1 NEED FOR THE STUDY

Education is recognized across the world as perhaps the most vital public service of all. However, the measurement of its impact is still in question. Assessments are essential to ensure that students get the educational support that they need to succeed, to see the effectiveness of different educational methods, and to ensure that education budgets are being utilized effectively. Therefore, inevitably, assessments also risk marking teachers, learners and institutions as successes or failures.

Various Assessments have been in existence as an informal tool for even longer than the actual annual school tests. The emphasis has been on ways in which an assessment can help to advance learning rather than merely measuring it. Current developments in education; such as the reinvention of assessment processes, suggest that the views of different experts on the correct and incorrect ways to assess educational progress are becoming more influential, and the research projects have helped to build on the evidence needed to improve assessments, and the educational outcomes which it is meant to focus on.

In India, independent assessments conducted by organisations like NCERT, ASER and Educational Initiatives show that students in India’s Government schools are not learning well, and not acquiring the expected competencies in basic literacy and numeracy.

EI has been carrying out research in learning outcomes for several years. It is a pleasure to acknowledge the Government of Himachal Pradesh and especially all officials of the Education Department for their contribution in advancing our ideas about appropriate ways of assessing educational outcomes.

Diagnostic scores are of increasing interest in educational testing due to their potential benefits, both remedial and instructional. Naturally, the number of educational tests that report diagnostic scores is on the rise across the globe, as are the number of researches based on such scores.

This report provides a critical evaluation of the diagnostic testing attempted by 28,898 students from classes 3, 5 and 7 across the state of Himachal Pradesh. It is demonstrated, using results from operational and simulated data, that diagnostic scores have to be based on a sufficient number of items and have to be sufficiently distinct from each other to be worth reporting and that several operationally reported sub-scores are actually not worth reporting. The format of the state-wide diagnostic testing has taken care of these basic elements.

A diagnostic test helps to identify a student's learning problems, so that teachers can provide specific instructions to remedy them. Moreover, teachers must often undertake complicated and time-consuming analyses of students' responses to individual items. Legitimate diagnostic tests permit teachers to use a test's results without having to devote hours to intricate interpretations.

This report on the Diagnostic Assessment in the state of Himachal Pradesh is a joint initiative undertaken by the Education Department – Govt. of Himachal Pradesh along with Sarva Shiksha Abhiyan (SSA) HP, Learning Links Foundation (LLF), Michael & Susan Dell Foundation (MSDF) and Educational Initiatives (EI).

1. DESIGN AND IMPLEMENTATION OF THE STUDY


7

1.2 ABOUT THE STUDY

In its first year of conducting this study, Educational Initiatives conducted diagnostic assessments for 1310 schools for class 3, 1246 schools for class 5 and 401 schools for class 7 (chosen as ‘sample’) across the 12 districts which fall under the jurisdiction of Education Department, Himachal Pradesh.

The tests were done for classes 3, 5 and 7 in Language (Hindi) and Maths.

Around 1000 students per class from each district were sampled and tested for classes 3, 5 and 7. Only classes with Hindi as the medium of instruction were covered in the diagnostic assessment. The sampling was performed by the State based on several criteria. It was ensured that in smaller districts almost all blocks are covered. In larger districts all blocks that are different in demography were covered to ensure a complete representation of the district. The clusters inside a block were chosen to represent different demographics of the block. First the class 7 schools were selected (these are secondary schools). Primary schools attached to secondary schools were selected post that. It was ensured that 1000 students from class 7 were selected. More primary schools were selected in same clusters to ensure 1000 students in each grade (3, 5). In some districts the total population is very less hence the sample size is low.

This report will argue that assessment must, in all cases, promote, rather than undermine, good education. Policy-makers need to keep the needs of pupils at the fore, and ensure that any evaluation of new developments in an assessment is carried out with careful consideration of the consequences, both intended and unintended.

It is helpful to make a distinction here between the intended uses of assessment data, and their actual uses. Assessments are often designed quite differently to ensure their adaptability for different purposes. We must therefore deliberate on the different ways in which such assessments are used. Paul Newton has identified 22 such uses2

1. Be part of effective planning

. These are, however, only broad categories. If one considers each purpose in detail, the number of uses for the data can multiply.

Similarly, the term “assessment for learning” is often interchangeably used as “formative assessment”. In 1999, the Assessment Reform Group defined assessment for learning as “the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there”. ARG also identified ten principles for formative assessment practice, arguing that it should:

2. Focus on how pupils learn

3. Be central to classroom practice

4. Be a key professional skill

5. Be sensitive and constructive

6. Foster motivation

7. Promote understanding of goals and criteria

8. Help learners know how to improve

9. Develop the capacity for self-assessment

10. Recognize all educational achievement

2 This list was submitted as part of the evidence from QCA to the inquiry on assessment and testing conducted by the House of Commons Children, Schools and Families Committee in 2007/08. See: http://www.publications.parliament.uk/pa/cm200708/cmselect/cmchilsch/169/16906.htm#n35


8

The persistent question thus is what is required to fulfill the aim of formative feedback; i.e., help learners to grow in capacity to manage their own learning, in order to progress effectively in developing capability in specific subject domains. As highlighted by the previous definition, learning can hence, be seen as a journey in which learners need to know where they are at present, where they are going, and how to get from one place to another.

The goals of the diagnostic assessment were to:

• Provide independent third-party assessment of student learning to be used by policymakers and researchers.

• Establish student learning in terms of knowledge and skills (or competencies) and provide feedback on the learning gaps, common errors and misconceptions.

• Provide analysis of student performance on specific competencies for classes tested.

• Analyze the learning patterns.

• Identify (if any) significant differences in performances across districts.

• Identify (if any) significant differences between the performance of boys and girls

• Diagnose the learning issues at the systemic level and provide pointers for further research.

In short, this diagnostic assessment report is a comprehensive study that aims to provide detailed question-wise and student-wise data for understanding and monitoring learning across classes.

1.3 SALIENT FEATURES OF THE STUDY

Coverage: This was a sample study covering around 1000 students per class per district from the government schools across 12 districts of Himachal Pradesh. All the students had Hindi as the medium of instructions. Details of the coverage of the assessment are given in the table alongside.

Scientifically Designed Test Development Cycle: The study included a detailed scientific process of test development supported by curriculum checking; development of test items based on the previous studies conducted by EI, fine-tuning and assembling of the final papers; detailed test adaptation, translation, validation of papers; master trainer workshops, field audit, data entry, and diagnostic data analysis prior to the release of the report.

Specially Constructed Papers: The test papers for the students were drawn from EI’s tests particularly designed for a wider pool of audience of the Michael and Susan Dell Foundation. These tests assess basic competencies as well as deeper conceptual understanding of the students through straightforward and higher order questions testing understanding and application of concepts, reasoning ability and critical thinking.

Benchmarking: In order to benchmark achievement of Himachal Pradesh students with those of Indian private schools (not low-cost schools) sufficient test items were used from ASSET3

3 ASSET is a diagnostic test from Educational Initiatives, which is administered every year to more than four lakh students studying in Indian private schools (not low-cost). Details can be obtained through www.ei-india.com.

.

District

Planned Sample

(as shared by the state)

Actual Sample*

BILASPUR 2,861 2,225

CHAMBA 3,816 2,904

HAMIRPUR 3,439 2,704

KANGRA 3,270 2,536

KINNAUR 791 319

KULLU 3,272 2,544

LAHUL & SPITI 785 501

MANDI 3,491 2,745

SHIMLA 3,474 2,606

SIRMAUR 3,773 2,699

SOLAN 3,403 2,266

UNA 3,704 2,712


9

Papers: For each class and subject, one set of papers was made and adapted into Hindi language. All schools got the same paper set for reasons of testing level and comparison across districts.

Specially Trained Test Administrators and Evaluators: A team of EI assessment experts conducted the training of Principals of District Institute of Education & Training (DIET) to roll out the test evaluators in their respective districts and to ensure standardized test administration. Test dates were decided by the State during the master trainers’ workshop after consulting the present DIET principals. It was decided that the master trainers would impart training to the teachers in their respective blocks and clusters on similar lines.

Conducting the tests: Test conduction was managed by the State of Himachal Pradesh with help from LLF. Printed papers were dispatched from Shimla to all the districts through DIET offices. The papers were completely written, however, the papers were read out aloud once for class 3. The teachers of the respective school invigilated the test after receiving appropriate trainings. The test timings were in line with the school timings of that area, however almost all the schools conducted the test on the same day.

Field Audits: EI teams carried out field audits during the testing day to check the quality and fairness of test administration in different schools. On the testing day, 31 schools spread across 5 districts were spot-visited by a team of 10 members. These 5 districts - Kangra, Kullu, Shimla, Sirmaur, Mandi - were chosen to cover parts of lower, central and upper Himachal. The team provided feedback to the test implementation and management team to ensure that the test administration was carried out smoothly. A report on the field audit was prepared by collating the feedback from all the team members and has been submitted as a part of this report in Chapter 4.

Data Entry: Students of class 3 filled in their answers on the question booklets itself. At class 5 and 7 level, the students directly scored their answers in the given OMR sheets. However, they answered the free response questions in the question booklets. After the test, the students’ responses (OMRs as well as question booklets) were sent back to EI where the class 3 complete question responses and class 5 and 7 free response questions were coded on OMR sheets by the evaluators at EI.

Analyses: Different types of analyses were carried out on the collected data to extract patterns in performances and to understand differences in learning levels across different groups. Advanced statistical methods were used to confirm patterns of learning. Distracter analysis enabled identification of misconceptions and common errors. The data was studied across different units of analysis: schools, districts, competencies, gender, etc. Tests of statistical significance were done to compare the performance of districts and girls and boys.

1.4 QUESTION PAPER DESIGN

Most education tests measure the status of students with respect to a cognitive skill or body of knowledge, which we can refer to as ‘assessed attributes’. The test needs to include a sufficient number of items to measure each assessed attribute so that the teacher can arrive at a reasonably accurate inference about how an individual student stands with regard to each of these attributes. For instance, if the teacher wants to know whether a student can multiply pairs of double-digit numbers, one or two items in a test won't provide an accurate estimate. The number of items required, of course, will depend on the nature of the skill or body of knowledge being measured, but one item per measured attribute definitely doesn't serve the purpose.

To arrive at a sound inference about student competency, the test should (1) measure a modest number of significant, high-priority cognitive skills or bodies of knowledge; (2) include enough items for each assessed attribute to enable teachers to make a reasonably accurate judgment on a test taker's mastery of that attribute; (3) describe with clarity what the test is assessing; and (4) not be too complicated or time-consuming.

The purpose of this assessment is to provide a detailed diagnostic snapshot on the strengths (students’ achievements) and weaknesses (learning gaps). The assessments were developed for Language and Maths for Classes 3, 5 and 7.


10

Paper Format: A simple, easy-to-administer test as well as its results may be better accepted by teachers and other stakeholders. This will also ensure that the findings of the assessments would be used to improve teaching and learning. The format of the test papers has been summarized below:

• The ‘written’ test had items that were read and answered by students themselves. However, all the questions that appear in the ‘written’ test for Class 3 were read out by the evaluators at the beginning of the test to familiarise the students with the questions.

• The Written test included Multiple Choice Questions, where the students chose an answer from four given options; and Free Response questions, where the student wrote down his/her answer in a fill-in-blanks format.

• The students in class 3 marked their answers on the question papers itself, and these responses were then recorded on the OMR sheet by a team of evaluation coders at EI. The students of class 5 and 7 recorded their own responses on the OMR sheet, but Free Response questions were answered on the question paper itself and similarly these questions were recorded on the OMR later.

• The invigilator manual that was provided along with the test papers. This had general and specific instructions on what needs to be read out to the students for each question in each test paper etc.

Competency Framework: The tests were comprehensive and included a number of competencies to ensure a more reliable measurement of student achievement. National Curriculum Framework documents, the Minimum Learning Levels (MLLs), the state textbooks, standard international framework of tests such as TIMSS and PIRLS, and EI’s large-scale benchmarking studies for Government schools, were referred to while finalizing the competencies for each paper. The tests were thus competency-based such that every question was linked to a particular competency in the skill framework or the test blueprint.

Types of Questions: The tests included questions to test not just knowledge (recall and procedure) but understanding and higher order skills. The focus of the questions was on testing for ‘learning with understanding’, i.e., the real understanding of students in concepts they have learnt in their specific classes. A percentage of questions were also drawn from EI’s national level benchmarking test, ASSET.

The following table provides details about the classes, subjects tested, paper length, duration of each paper, the proportion of free response (FR) and multiple choice questions (MCQs) and the proportion of straight forward (SF) and higher order (HO) questions:

Subject Class # Questions # SF Questions # HO Questions

Duration # FR Questions # MCQs # FR

Questions # MCQs

Maths

3 26 -- 13 -- 13 90 mins

5 32 -- 16 -- 16 105 mins

7 36 -- 18 -- 18 105 mins

Language (Hindi)

3 32 8 8 1 15 105 mins

5 38 2 16 5 15 105 mins

7 44 2 20 3 19 105 mins


11

Sample Questions

A. Questions that check for learning that is straightforward or text-bookish

Generally, it is believed that though the students may not be learning with conceptual understanding, they perform well on questions which test concepts in a format that is familiar to them. This can be tested by placing questions which are similar to examples or questions of text books. They are described as ‘straight forward’ or ‘text-bookish’ questions.

B. Higher Order Questions - that check for a deeper understanding of concepts:

While students perform well on easy and familiar questions, these questions check for a conceptual understanding of the subject in question and whether students can apply what they have learnt in different formats.

C. Higher Order Questions - that check for application of concepts

These types of questions checks the student’s ability to answer the questions based reasoning, visual interpretation, and estimation. These are the kind of questions which students are faced with in their day-to-day interaction with the world.

Maths Class 3

Hindi Class 5

Maths Class 7

Maths Class 3

Maths Class 7


12

D. Questions that check for Reading Comprehension

Several questions in the Language papers are based on ‘unseen’ passages and authentic material seen in daily life such as tables, labels, notices, etc. These passages test comprehension which is both fact based and implied in the text.

Scoring Process: At EI, test papers for every class are supplemented with a ‘scoring card’. The scoring card provided a question-wise rubric (with scores / codes / answers) for how each response is to be scored. The scoring card captured common mistakes that children tend to make while answering questions. The data that was collected by this process was critical for the diagnostic feedback and analyses. This form of ‘scoring’ is common international practice in tests such as TIMSS.

Maths Class 5

Maths Class 7


13

The evaluators from EI for class 3 captured the answer responses of the students in the form of codes as specified in the scoring card and entered them on to ICR and OMR sheet. The codes thus entered were scanned using ‘intelligent character recognition’ (ICR) and ‘optical mark recognition’ techniques and was converted into a database of student responses for further analysis. This eliminated human error while transcribing, etc., and also saved time. While coding responses for ‘multiple choice’ items, only the ‘option letter’ ticked was to be recorded. For ‘free response’ items, the responses were recorded based on the detailed scoring rubrics. All scoring cards also had general codes assigned for specific types of responses common to all papers. The details are given below:

General Codes Assigned Across Papers:

1. For questions with options A, B, C, D: Option ticked (enter option A, B, C, and D in English).

2. Items that have not been attempted at all should be coded as 88.

3. Invalid answers (e.g. question copied, more than one option ticked in multiple choice questions, crossed out answers, illegible answers) should be coded as 86.

4. Spellings are not to be checked unless indicated specifically.

5. Instead of ticking, if a child crosses or circles or marks an option in any other way, or rewrites one of the options, that option should be treated as their choice.

6. Free Response Questions were coded as follows:

a. Correct Answer - 01

b. Partly Correct Answer- codes vary with every question - 11, 21, 31 (depending on the type of FR question)

c. Any other answer - 85

Sample Test Items and Their Scoring Rubrics:

Class 7 Maths - MCQ Question

Option ticked (enter option A,B,C,D in English) A

B

C

D

Invalid Answer Code/More Than One Option Ticked 86

Not Attempted 88

Maths Class 7


14

Class 5 Language- Free Response Question

Meaning comes out clearly; no grammatical errors; sentence complete and related to the given picture 1

Meaning comes out clearly; 1-2 grammatical errors; sentence complete and related to the given picture 11

Meaning not completely clear; some grammatical errors; sentence incomplete but related to the given picture 21

Meaning not clear at all but 1-2 appropriate words related to the given picture are used 31

Sentence not related to the given picture 81

Any other answer 85

Invalid Answer Code/More Than One Option Ticked 86

Not Attempted 88

Language Class 5


15

2.1 PERFORMANCE ACROSS SUBJECTS AND CLASSES

The overall performance for each class and subject is presented in the table below. The state average performance ranges from 48.5% to 59.5% in Hindi and 38.1% to 59.7% in Maths across the three tested classes. It was observed that the performance was more homogenous within higher classes, indicated by lower standard deviation.

Subject Class N

(Total students) % Score

(Average) Std. Dev.

Language (Hindi) 3 8,231 59.5 % 21.4 5 9,151 59.0 % 18.4 7 9,516 48.5 % 15.6

Maths 3 8,231 59.7 % 24.3 5 9,151 48.2 % 20.6 7 9,516 38.1 % 18.4

While the overall performance in Hindi has been largely consistent in Classes 3 and 5, the dip in performance in Hindi class 7 and the decline in overall performance across classes in Maths cannot be explained through an analysis of the average scores. A comparative analysis may not suffice for this purpose, and a system of benchmarks against which the scores can be analysed is essential. A question-wise analysis is also essential to understand what particular skills and competencies the students are not performing well in, and whether the decline in the overall performance can be traced to weaker conceptual understanding in particular skills.

2.2 COMPARATIVE PERFORMANCE AGAINST INDIAN PRIVATE SCHOOLS

A comparative analysis was undertaken between the performance of Himachal Pradesh Students and the performance of Indian private schools as represented by ASSET (A diagnostic benchmarking national level test) in English, Maths, Science, Hindi and Social Science. Himachal Pradesh students typically score lower than Indian Private Schools.

2. MAIN FINDINGS


16

Sufficient common questions4

(7 -11 questions in each paper) were used from ASSET to compare the performance of Himachal Pradesh students with English medium private schools. Gap between Himachal Pradesh students and students of Indian Private Schools was found to be comparatively high in Class 7 and lower for Class 3 (Hindi and Maths). A closer look at Class 3 questions reveals that Himachal Pradesh students have performed better than Indian Private School students on certain questions. In Language, for all the classes those questions wherein students had to infer from the given passages/ stories/ notices, were found to be low performing compared to those questions which had answers directly stated in the passages/ stories/ notices. The examples below illustrate the difference seen between Himachal Pradesh schools and Indian private schools on common ASSET questions.

2.3 LEARNING WITH UNDERSTANDING

A high level analysis of questions which tested higher order (HO) skills as opposed to questions which were straight forward (SF) reveals that students have performed better on SF questions. Broad trend on these lines also indicates that performance on HO questions goes down as we move higher classes.

4 25% ASSET items used for the analysis

Maths Class 7

Maths Class 3

Sample Question: This question tests students’ understanding of operations and the relation between them. 50% of HP students answered this question correctly compared to 46% of ASSET students.

Sample Question: This question checks if students are able to observe a pattern and logically think of a general rule being followed in that pattern. Only 29% of HP students answered this question correctly compared to 49% of ASSET students.


17

Deeper analysis of question-wise performance of students highlights that learning with understanding is weaker in higher classes for questions testing conceptual learning. This indicates that the conceptual understanding in specific topics is not deep enough for the student to understand and attempt questions that are presented in a non-familiar format or which go beyond the textbook. Rote-based learning among the students indicates that conceptual understanding is not entrenched enough for the student to progress to higher order questions and skills which require a strong basic understanding of the concept. This has been explained though the examples given below.

This question tests the students’ understanding of basic shapes. Only half of the students of class 5 have been able to correctly identify the rectangle among the given shapes. About 28% of students have chosen option C, a rounded rectangle. This indicates that these students are not able to recognise a rectangle in a non-standard orientation. They may have a mental picture of a rectangle in a standard orientation, and may have chosen the image that is closest to this picture, irrespective of whether it satisfies all the properties of a rectangle or not.

Maths Class 5


18

This question tests the understanding of place value system in decimals. Students are expected to realise that 7 tenths and 8 thousandths make 0.708. Only about 24% of students have been able to answer this correctly. Students may be used to converting conventional expanded forms like 7/10 + 0/100 + 8/ 1000 into decimals. But here when the conventional order is changed and hundredths are missing, many students may have been confused. About 34% have chosen option C, 0.8007 and 15% have answered D, 0.87, retaining the order in which the numbers appear in the question.

Maths Class 7


19

2.4 PERFORMANCE ON PRACTICAL COMPETENCIES

Practical competencies are defined as visual interpretation, map reading, observation and estimation- all of which are essential skills for daily life and for the overall development of the student, irrespective of the level of education that student pursues in the future. Based on the student performance on questions testing these skills, it was observed that the overall performance is low in these questions. An example of a question testing practical competencies is given below:

This question checks if students are able to estimate the length of an object relative to another given object. Only 38% students have answered this question correctly. Around 30% students have chosen the wrong answer option A. They have probably just looked at the height of the scale given in the image and chosen 100 cm as their answer.

For more detailed analysis by skill and question, please refer to Chapter 4 of this report. The skills, questions have also been analysed at a district level to understand the strengths and weaknesses in competencies for a particular district. A district wise analysis report for each district is also shared with this report.

Maths Class 5


20

2.5 PERFORMANCE ACROSS DISTRICTS

2.5.1 District Performance on Composite Performance Index

To have a comprehensive understanding of the state’s performance at all levels, a district wise analysis was performed and relative performance of the districts were analysed. A Composite Performance Index was calculated for all districts, which is a simple average of performance across classes and subjects. This composite index was calculated for all districts and their scores on this index were analysed.

At the same time, an analysis only based on an overall performance may be insufficient as well as misleading; a particular district may be the top performer, but may have scores very close to the average. To incorporate differences of the district scores from the state average, effect sizes were calculated for each district through Cohen’s D5

Based on the district wise performance on this index and their effect sizes, a few observations have been presented.

, which would highlight if the district’s performance is significantly higher or lower than the state average.

• Hamirpur outperformed all districts, with a score significantly higher than the state average.

• Chamba, Bilaspur and Sirmaur were among the bottom three districts when compared to state average

• Kinnaur and Lahul & Spiti removed from the district-wise analysis due to low sample size for both the districts

• The remaining districts of Mandi, Kangra, Kullu, Solan, Una and Shimla performed close to the state average score.

5 Cohen's d is defined as the difference between two means divided by a standard deviation for the data


21

2.5.2 District Performance across classes and subjects

The composite performance index was also calculated separately for each subject to analyse if the performances of the districts vary between the two subjects. The difference of this score from the state average was also calculated through the Cohen’s D.

Based on the district wise performance on the index in Hindi and Maths, observations have been presented below

• Hamirpur was the top performer in both subjects, performing significantly better than the state average

• Mandi performed significantly better than other districts and the state average in Maths, but performed close to average in Hindi, leading to a decline in its overall performance.

For a detailed view on each district, please refer to the district wise report (provided for every district) along with this report.

Math Hindi


22

2.6. PERFORMANCE OF BOYS AND GIRLS

Over 13 thousand girls and a similar number of boys participated in the study. This section of report attempts to analyse any noticeable difference between the average performance of the boys and girls in the study. The table below indicates the mean difference in the performance according to gender.

Class Subject Girl Boys T

Critic value

Cohen's d

Effect Size- interpretation N Avg SD N Avg SD

3 Hindi 4189 60.5 21.2 3956 58.4 21.4 4.4 0.1 - 5 Hindi 4409 59.6 18.4 4716 58.6 18.3 2.6 0.05 - 7 Hindi 4682 49.4 15.8 4830 47.6 15.3 5.6 0.12 - 3 Maths 4189 60.6 24.3 3956 58.6 24.3 3.7 0.08 - 5 Maths 4409 48.6 20.6 4716 48 20.6 1.4 0.03 - 7 Maths 4682 38 18.4 4830 38.1 18.4 -0.3 -0.01 -

On comparing performances according to gender, it was observed that girls perform better than boys in Hindi class 3, 5 and 7 and Maths class 3 but the magnitude of difference was marginal for all classes. In Maths class 5 and 7, both boys and girls performed similarly.

Legend

Boys and girls are performing at the same level.

Girls are performing significantly better than boys with <0.1 effect size.


23

3.1 MISCONCEPTIONS AND COMMON ERRORS

Students develop a cognitive understanding of the world around them through interactions based on their daily experiences. Teachers and schools help build this understanding. Misconceptions are concepts that students acquire which are not in line with or do not match the conventional expert view in that topic. These result in cognitive gaps in their understanding. Identifying the exact nature of student misconceptions is difficult through regular classroom interactions. Any question in an assessment test that attempts to identify the misconceptions of the students must force the students to actively use their conceptual understanding. A detailed understanding of these misconceptions through large scale diagnostic assessments provides the teacher with a starting point to explore these in the classroom and eventually help to build correct notions or conceptual understanding. Detailed analysis of the answers given by students in each question in all the subjects tested reveal that students’ understanding of the concepts is often faulty as they seem to harbor many mistaken notions. They also seem to commonly make errors in answering some questions. Identifying the exact nature of student misconceptions is difficult through regular classroom interactions. Any question in an assessment test that attempts to identify the misconceptions of the students must force the students. Some misconceptions and common errors found in the analysis of HP state study include:

• The bigger the area of a shape, the larger will be its perimeter

• Confusing Part-Whole relationship with Part-Part relationship in Fractions and Ratios

• Integers can be added without checking their positive/ negative signs

• Using prior knowledge to answer reading comprehension questions

An item wise analysis of all the questions reveals these patterns through a graphical representation known as Item Response Curve (IRC). A short description on how to read an IRC is given below.

How to read the graph:

These are the item response curves for a question. The X axis is the total score in the paper and the Y axis is the percentage of students. The graphs show what percentage of students scoring different total scores in each class, chose which answer options for the question. The legend box shows the total percentage of students who chose each option in the paper. In this graph, the percentage of students choosing option A increases as one moves from a low total score to a high total score, while percentage of students choosing option D increases up to a certain total score and then starts reducing.

y

x

3. INSIGHTS ON SPECIFIC LEARNING AREAS


24

3.1.1 Misconceptions and Common Errors in Mathematics

Question Graphs

Class 3 / Question 15

Explanation: This question checks if students can understand how a paper would look like when it is folded, cut and then unfolded. Students are expected to visualize the shape that would be formed after the paper is unfolded. Only 30% students chose the correct answer (D). However, a lot of students chose option C (19%) and were probably just looking at the image in the question and choosing this option, not realizing that when the paper is folded into half and cut, double the part of the paper will be cut. Around 35% students have not attempted this question.


Explanation: This question tests the understanding of place value. About 50% of students have answered it correctly as option A, and 22% have answered it as C. These students may have mechanically put down the digits in the order in which they appear in the question. They may not have noticed that the question asks for 3 ones and 7 tens. This could be because they are used to solving problems in the conventional order – 7 tens and 3 ones and are not sufficiently exposed to equivalent alternate forms of writing the expanded forms of a number.

x

y


25

Question Graphs


Explanation: This question checks if students are able to represent a part of a group as a fraction. Students are expected to understand that 10 students out of total 40 (30 girls + 10 boys) are boys and hence the fraction of boys would be ¼. However a lot of students have chosen option A (25%) and option C (39%). Students choosing option A probably are not considering the fact that the total number of students is 40 and are calculating the ratio of boys to girls rather than the fraction of boys in the class. Students choosing option C are probably making the same error but are giving the ratio of girls to boys. These students have not been able to differentiate a part-whole relationship from a part-part relationship.


Explanation: This question checks if students can apply fractions to the units of capacity. Students are expected to understand that a litre has 1000 millilitres and ¼ litre would be equal 250 millilitres. Only 6% students have chosen the correct answer option A. However, most of the students (41%) have chosen option B. These students seem to have the understanding of fractions but are probably choosing this option as ¼ part of the jug is filled here. Students choosing option C (31%) may be matching the 4 of ¼ with 400 ml. They do not seem to understand the concept of fractions.


26

Question Graphs


Explanation: This question tests the understanding of perimeter as the length of the boundary of a shape. Students are expected to identify the shape that has the longest boundary. But almost 31% of students have answered D. Possibly they are not able to differentiate between the terms area and perimeter, and have chosen the piece that has the largest area. It is also possible that they have a notion that the shape with the largest area has the largest perimeter too.


Explanation: This question tests the students’ understanding of basic shapes. Only half of the students have been able to correctly identify the rectangle among the given shapes. About 28% of students have chosen option C, a rounded rectangle. This indicates that these students are not able to recognize a rectangle in a non-standard orientation. They may have a mental picture of a rectangle in a standard orientation, and may have chosen the image that is closest to this picture, irrespective of whether it satisfies all the properties of a rectangle or not.


27

Question Graphs


Explanation: This question checks if the students can read a pictograph and answer questions based on it. The students are expected to understand from the key that since the picture of one book stands for 2 books, Reena read total 32 books in her vacation. Only 27% of the students chose the correct answer. Students choosing option C (49%) have probably not understood the key or not used it at all. They have just counted the number of books and chosen this option.


Explanation: This question checks if students are able to estimate the length of an object relative to another given object. Only 38% students have answered this question correctly. Around 30% students have chosen the wrong answer option A. They have probably just looked at the height of the scale given in the image and chosen 100 cm as their answer.


28

Question Graphs


Explanation: This question checks if students are able to logically solve a problem based on a daily life context. Students are expected to understand that they need 3 more marbles for equal distribution and look for a number which would become divisible by 10 on adding 3. Only 32% students have answered this question correctly. An equal percentage of students (32%) have chosen the wrong answer option A. They probably have not understood the question and have just added the numbers given in the question.


Explanation: This question checks if students are able to find the sum of given integers. Only 36% students have answered this question correctly. A lot of students (31%) have chosen the wrong answer option A. They probably have added all the 3 numbers and used the negative sign with the answer. This indicates that the concept of integers/addition or subtraction of integers is not clear to them.


29

Question Graphs


Explanation: This question checks if students are able to find the perimeter of a given figure. They are expected to understand that the side of each small square on the grid is 1 cm and then calculate the perimeter based on that. Only 23% students have chosen the correct answer option C. A lot of students (41%) have chosen option A. They have probably either just counted the number of small squares that are shaded or have found out the area instead of the perimeter. One reason for this could be that traditionally such grids in the textbooks are used to find out the areas and not perimeters and students are also confused between the two concepts – area and perimeter.


Explanation: This question checks if students are able to add two fractions with different denominators when given in a real life context. Only 32% students have answered this question correctly. A lot of students (34%) have chosen option D. They have probably just added the numerators and denominators separately.


30

Question Graphs


Explanation: This question checks if students are able to simplify a simple algebraic expression. Students are expected to know that when the brackets are opened the sign (- or +) and the digit outside the brackets get multiplied with each term inside the brackets. Only 20% students have answered this question correctly. Students choosing option A (36%) have probably not multiplied 5 with the ‘y’ inside the brackets. Students choosing option C (20%) have probably multiplied 5 with the ‘y’ inside the brackets but ignored the ‘-‘ sign.


Explanation: This question tests the understanding of place value system in decimals. Students are expected to realize that 7 tenths and 8 thousandths make 0.708. Only about 24% of students have been able to answer this correctly. Students may be used to converting conventional expanded forms like 7/10 + 0/100 + 8/ 1000 into decimals. But here when the conventional order is changed and hundredths are missing, many students may have been confused. About 34% have chosen option C, 0.8007 and 15% have answered D, 0.87, retaining the order in which the numbers appear in the question.


31

Question Graphs


Explanation: This question checks if students are able to observe a pattern and logically think of a general rule being followed in that pattern. Students are expected to understand that 2 corners in each face will not have antennae and hence a face having n corners will have n-2 antennae. Only 29% students have answered this question correctly. A lot of students (31%) have chosen the wrong option C. They probably have thought that the 50th figure will have 50 antennae but not considered the fact that the face having 50 corners will not be the 50th figure.


Explanation: This question checks if students are able to express a part of a group in terms of ratio. Students are expected to calculate the number of boys in the class by subtracting the number of girls from the total number of students and then give the ratio of number of boys to number of girls. Only 25% students have answered this question correctly. A lot of students (33%) have chosen the wrong answer option D. They are either just looking at the numbers given in the question, i.e., 30 and 20, and are choosing their answer as 3:2 or are not careful about what is being asked in the question.


32

3.1.2 Misconceptions and Common Errors in Language

Question Graphs


Explanation: The question tests students’ logical analysis of information in a poster. The poster is about a contest where students have to send hilarious responses to the questions asked. The judges will have to read the responses before announcing the winner. Students who have understood this have chosen the correct answer, option B. Most students (over 50%) have chosen to skip answering the question probably because they were looking for direct facts mentioned in the poster. It is also possible that due to long length of answer options students had difficulty reading and therefore skipped this one. It is likely that students who chose options A, C, or D relied on guesswork.

Class 3/ Question 22

Explanation: This question tests students understanding of words and their meanings based on the context. In this question, the

word tested is ‘snatched’ (झपट ल�ं) and the

answer is A: ‘took quickly’. Some students have understood this in the context and have chosen option A. 32% students have processed the information a little further and have chosen option C, ‘ate fast’, which is what the goats may do after snatching, but that is not the act of snatching. These students may either have not understood the question or the context, or may have been short of time.


33

Question Graphs


Explanation: This question tests students’ understanding of explicit and implicit information presented in a given text. In this question, there is a mention of the first three prizes and then there is a mention of additional prizes being given away. Students who have got the correct count after reading the authentic text have chosen the correct answer, option D. Students who chose options B or C, (56%), as their answer have missed out reading the details in the passage. It is also likely that those who chose option B have been influenced by the prior knowledge that there are generally only three prizes given away.


Explanation: This question tests students’ ability to identify the main theme in a given text. In this context, the main activity as required by the poster is to complete a dialogue. Students who have been able to abstract this information have chosen the correct answer, option D. Many students (35%) have misunderstood the question and have chosen option C, ‘use the pen or buy it’. They probably have thought that the main activity the boy in the comic is doing, is being asked about. Those choosing other answers are merely referring to details in the passage.


34

Question Graphs


Explanation: This question aims to test if student understand implied facts. In the question, the line is written in a certain manner to imply that it is not the norm and is in fact the opposite of what generally happens. 37% students have incorrectly chosen option D which is very similar to the correct answer (A) but is not exactly what it means. This shows that they have either resorted to prior knowledge or have not understood what the sentence means. Those choosing other options have tried to locate facts from the passage to supply as the answer.


Explanation: This question aims to test if student understand implied facts. In the question, the meaning of a phrase is being tested in context of the story. Majority of students (38%) have wrongly chosen option B. This could be because of they fact that they didn’t understand the meaning and chose the answer which seems like a genuinely good quality generally seen in the protagonist of a text. Those choosing other options are basing their responses on either what they themselved would do, or what they assumed the protagonist may do in such a situation.


35

Question Graphs


Explanation: In the question, the traits of a character based on the context in the text are being tested. Most students (43%) have failed to understand this and have based their answer on the text that follows this context, choosing option A which is an explicit detail as the answer over an implicit one. Those choosing options B and C have either not understood the detail being tested and have picked a response based on the overview of the context, or they have used prior knowledge based on such stories where the antagonist has negative qualities and tries to get the protagonist in trouble.


Explanation: This question aims to test if student can recall stated facts. The tested sentence is very clearly mentioned in the passage. Only 20% students have been able to recall this fact correctly and have chosen the correct answer, option B. Students who have chosen the wrong answers have done so either because they were trying to locate implicit details or because they were looking at other elements of the situation in the passage.


36

Question Graphs


Explanation: In this question, the underlying meaning of a given statement is being tested. Only 21% students have understood the overall text and have been able to derive the meaning by choosing the correct answer, option A. Students choosing other options are looking for clues in the preceding or following content.


Explanation: This question aims to test students’ understanding of the text and the references made in certain places. Here, the focus of the word is on the ‘incident’, option D. Most students (44%) have chosen option A, referring to the person with whom the incident happened rather than the incident itself. They have failed to understand that the reference is being made to the case and not the person involved. Some students have chosen option B, wrongly assuming that the reference is being made to the knife lodged in the patient.


37

Question Graphs


Explanation: This question tests students understanding of the text by being able to differentiate between facts and opinions. The given passage is in the form of a story. The phrase is the writer’s opinion about what happened. Students who have chosen option A (20%) have probably not understood the phrase and have assumed it to be a cause and effect question. They have chosen an answer which tells us that nothing really happens to the protagonist despite his errors. 30% students chosen option C, probably because they have not understood the phrase tested and have gone for something which relates to both Vasco and Malindi.


Explanation: This question tests student’s ability to draw inferences from a story. In the given passage, it is Meena who thinks she is a brave girl as she chooses the brave thing to do to overcome her fears. 24% students have chosen option A; this could either be due to prior knowledge that parents always think that their children are brave and are the best. It could also be because they assumed that she may have later told her mother that she faced the situation bravely and her mother must have called her a brave girl. Students who chose option B must have done so thinking along the similar lines as those who chose option A, only replacing mother with the music teacher. Students who chose C probably thought that the teacher thinks is brave as she overcomes her fears and writes down the bravest thing she will do. Also, in the end it is mentioned that the teacher praises her, though not for the same reason.


38

3.2 PERFORMANCE IN DIFFERENT SKILLS

Analysis was carried out on the performance of the students in the various skills across different classes and subjects. This analysis enables one to identify patterns in performance across the different skills and understand which skills are the strongest and which are the weakest. This understanding will pave the way for appropriate action in the classroom.

3.2.1 Skill-wise Performance in Hindi

Class 3

S. No.

Medium Class Subject Skill No. of

Questions Sample

Size %

Score

1 Hindi 3 Language Recognises and writes letters and knows starting sound of words 3 8231 85.3

2 Hindi 3 Language

Reads and writes simple words that are moderately difficult and have 3-4 letters and knows names of objects, birds, animals, etc. seen in daily life

4 8231 81.7

3 Hindi 3 Language Uses words appropriate to the context based on their meanings, number and gender

5 8231 75.6

4 Hindi 3 Language Reads and writes 3 simple and short sentences that have less than 5 words in a sentence

5 8231 52.6

5 Hindi 3 Language Reads descriptive text, of 3-5 sentences independently and comprehends beyond stated facts

12 8231 48.2

6 Hindi 3 Language

Understands written information presented in various forms as Tables, Notices, Tickets, Posters, Labels, etc seen in real life

3 8231 34.0

High Performing Skills Low Performing Skills

Class 5

S. No.


Questions Sample

Size %

Score

1 Hindi 5 Language Reads and writes simple words (that are class level appropriate) and knows names of objects, birds, animals, etc seen in daily life

5 9151 81.3

2 Hindi 5 Language Reads and writes sentences using grammar concepts

8 9151 68.2

3 Hindi 5 Language Knows synonyms, antonyms and deduces word meanings from clues in context

4 9151 70.6

4 Hindi 5 Language Reads descriptive text and comprehends explicit and implicit details for class level

16 9151 53.9

5 Hindi 5 Language

Understands written information presented in various forms as Tables, Notices, Tickets, Posters, Labels, etc seen in real life for class level

5 9151 29.3



39

Class 7

S. No.


Questions Sample

Size %

Score

1 Hindi 7 Language Reads and writes simple words (that are class level appropriate) and knows names of objects, birds, animals, etc seen in daily life

4 9516 71.9

2 Hindi 7 Language Reads and writes sentences using grammar concepts 8 9516 65

3 Hindi 7 Language Knows synonyms, antonyms and deduces word meanings from clues in context 3 9516 56.9

4 Hindi 7 Language Reads descriptive text and comprehends explicit and implicit details for class level

20 9516 38.6

5 Hindi 7 Language

Understands written information presented in various forms as Tables, Notices, Tickets, Posters, Labels, etc seen in real life for class level

9 9516 42.4


• In Hindi language, the skill of Recognizes and writes letters and knows the starting sound of words observed the highest performance in class 3 while Reads and writes sentences using grammar concepts was the highest performing in classes 5 and 7

• The weakest skills in all classes in Hindi were skills of reading comprehension: Understands written information presented in various forms as Tables, Notices, Tickets, Posters, Labels, etc. seen in real life and Reads descriptive text and comprehends explicit and implicit details for class level. Thus understanding both the stories as well as authentic text material seems to be a challenge.

• Students seem to be weak in reading comprehension beyond stated facts. Performance is poorer on inference based questions as compared to straight forward – fact based questions. Examples given below elucidate the same.

• Students should be encouraged to write their own stories, as well as discuss stories that they have read. Students can be asked to list the different qualities for each character in the story and describe how they feel the characters will react in different situations and contexts.

Hindi Class 7

Hindi Class 7

Sample Question: Q31 is a straight forward question testing a fact cited in the passage. Student performance on this question is 65.7%.

Sample Question: Q36 requires students to infer after reading the entire story, keeping in mind the facts stated. Student performance on this question is 17.1%


40

3.2.2 Skill-wise Performance in Maths

Class 3

S. No.


Questions Sample

Size %

Score

1 Hindi 3 Maths Pre-maths skills 3 8231 78.9

2 Hindi 3 Maths Number sense and basic number competency 5 8231 57.5

3 Hindi 3 Maths Arithmetic operations: addition and subtraction

4 8231 55.7

4 Hindi 3 Maths Arithmetic operations: multiplication 3 8231 67.3

5 Hindi 3 Maths Basic shapes and geometry 4 8231 59.6

6 Hindi 3 Maths Application in daily life: money, time, calendar, length etc.

4 8231 52.2

7 Hindi 3 Maths Problem Solving 3 8231 51.8


Class 5

S. No.


Questions Sample

Size %

Score


2 Hindi 5 Maths Four arithmetic operations 5 9151 56.3

3 Hindi 5 Maths Fractions: concepts and applications 4 9151 40.5

4 Hindi 5 Maths Application in daily life: money, time, calendar, length etc.

5 9151 41.1

5 Hindi 5 Maths Basic shapes and geometry 6 9151 51.5

6 Hindi 5 Maths Data interpretation and analysis 4 9151 54.2

7 Hindi 5 Maths Problem Solving 4 9151 38.3


Class 7

S. No.


Questions Sample

Size %

Score


2 Hindi 7 Maths Factors and Multiples 3 9516 37.8

3 Hindi 7 Maths Application in daily life and word/visual problems

3 9516 33.7

4 Hindi 7 Maths Area and Perimeter 4 9516 37.3

5 Hindi 7 Maths Geometry: concepts and application 4 9516 45.1

6 Hindi 7 Maths Algebra: concepts and application 3 9516 35.8

7 Hindi 7 Maths Fractions, Decimals and Ratios: concepts and application

4 9516 42.2

8 Hindi 7

Problem Solving 4 9516 27.8

9 Hindi 7

Data interpretation and analysis 4 9516 46.5

10 Hindi 7

Integers: concepts and application 4 9516 33.7



41

• In Maths, the strongest skills were Pre-Maths skills and Multiplication in class 3, four basic arithmetic operations in class 5 and Data Interpretation and Analysis and Geometry in class 7.

• The weakest skills were Applications in daily life and Problem Solving across the three classes.

• Transition from basic skills in primary classes to intermediate skills like Integers, Algebra in higher classes seems to be a challenge

o Students of class 7 seem to be struggling with Area and Perimeter problems – a concept that is introduced in Class 4

o Integers, a concept introduced in Class 5, also seems particularly weak as many students are unable to answer straight forward questions

o The transition from concrete to abstract concepts in maths is a challenge and can be systematically addressed through targeted curriculum efforts

• Within a skill, different kinds of questions are covered - some could be straightforward or procedural, some could be conceptual and some could require higher order thinking. In skills found weak, students are not able to answer the conceptual and higher order thinking questions. This could be because of the stress laid on the procedural ways of answering or solving questions. If instead, stress is laid on conceptual understanding, students are more likely to answer different kinds of questions correctly.


42

EI conducted field audits as a part of this assessment. Over 30 schools were visited during test conduction by 10 Field Audit Resource Persons. The purpose of the audit was to ensure that the execution of the test met certain standards of quality, maintained a certain level of standardization and to assess whether any issues of bias or subjectivity can be traced at any of the schools. To ensure that the observations are comprehensive, the auditors not only observed the Test Administration, but also interacted with the Teachers, Head Teachers and Students to take feedback and opinions about the test and the testing process.

Each auditor spent an average of 40 minutes in each school and recorded his/her observations on a detailed checklist. The key observations which the auditors were asked to make in their visits included the adherence to the processes given in the evaluator manual, whether the teacher had participated in the evaluator training, infrastructural observations such as issues of disturbance and cluttered classrooms, the responsiveness of the teachers to the tests and testing processes, issues of cheating and bias through observations on seating arrangements and teacher involvement, and perception of the teachers and students regarding the test and the testing process. The responses were then analysed to deduce some key test conduction practices and observations. Some key observations from the field audits are discussed below:

1. All the schools audited (except one) were ready with the test material and started the test at the scheduled time. Test duration and timings were followed in all the audited schools. Seating arrangement was proper and students were sitting comfortably. In most of the audited schools, students from an individual class sat separately from other classes. No instance of mass assistance observed in the audited schools.

2. Over 80% of the evaluators present during test conduction did not get any formal training from the master trainers who were trained by EI for this purpose. Though most of them were aware of the broad procedures; in some schools this caused a little confusion and needed intervention by the Resource Persons in the classroom from time to time.

3. The processes from the evaluator manual were not consistently followed in all the schools. This can also be associated with the unfamiliarity of the evaluators with the processes indicated in the Manual.

4. Apart from the evaluator, there were other people present in the classrooms in around 40% of the observed schools. The presence of external persons during test administration can have certain repercussions of bias and unnatural environments in the classroom.

5. Over 80% of the school evaluators were aware of the school codes for their schools and helped the students write the code on their sheets.

6. Over 80% of the classes observed conducted the tests with no serious disturbances.

7. Almost all the schools observed practiced seating arrangements that ensured that the room was not overcrowded and that the students were seated in rows or a circle; in a way that ensured that the arrangement minimised the issues of cheating and were deemed appropriate for a testing environment.

8. Most of the evaluators had enough test material for all students and for themselves to conduct the test according to the procedures. However, in four of the observed schools, testing material was not sufficient for all the students to test independently. Issues of shortage included the unavailability of the paper for the teacher

4. FIELD AUDITS


43

to read out the questions for class 3, exclusion of some students due to unavailability of papers as the enrolment data shared was for previous year.

9. In a few schools, (about 26%) there were a few instances noticed where the teachers were assisting the students in answering the questions, usually by giving clues.

10. In one school, the test was not conducted during the Resource Person’s visit. Since the conduction of the test was supposed to be at the same time across the schools, this school was probably not informed about the testing.

11. With respect to the time durations of the tests, most schools followed the time durations prescribed for the test. Issues of time included delayed commencement of the test and confusion and unfamiliarity about the test practices such as School Codes and OMRs.

12. Some of the schools reported multi-grade classrooms. While some of the classrooms were effectively partitioned to ensure minimum disturbance, others were more chaotic and not suitable for a testing environment. However, no significant patterns were observed with respect to the classroom structure.

13. Most students shared that they found the test interesting and mentioned a few questions / passages form the test which were fun to do / read.

14. Some of the students and teachers reported that the testing format was unfamiliar and difficult, while others commented on the difficulty levels of the Hindi papers and Reading Comprehension sections. This observation however, varied across the interactions with the students and teachers.

The observations from the field audit visits have been summarized in the table given below.


44

School Code

Did the teacher attend the evaluator training?

Were processes from the Evaluator Manual

followed?

Were there other

people present during testing

?

Did the teacher

know the

school code?

Did she help

students write

it?

Disturbance in class?

Was the Class

overcrowded?

Was the seating

arrangement APPROPRIATE for testing?

Did the teacher

have sufficient

test material?

Was time

schedule and

duration followed

?

Comments

2110907702 N Y Y Y Y Y N Y N Big hall where classes were being conducted simultaneously, disturbance, friendly teachers, teachers said that the paper was tough

2110907701 N N Y Y N N Y Y Y Students and teachers both found the tests very difficult 21109079011 N N Y Y N N Y N N Teacher filled OMR sheet himself

2020602602 N N Y Y Y N Y Y Y Teachers on rotation, thought language paper tested facts, given no formal training, shortage of papers

2020602601 N N N Y N N Y Y Y No formal training for teachers, shortage of papers

2020601402 N N N Y N N Y Y Y No formal training for teachers, shortage of papers, did not find papers tough

2110904801 N Y N N N N Y Y Y 2110904802 Y Y Y NA N N Y Y Y 2110904501 Y Y N Y N N Y Y Y Teachers and students liked tests 2110904501 N Y N Y N N Y Y Y Teachers and students liked tests 2040304307 N Y Y Y N N Y Y Y

2040301802 N N Y Y N N Y Y Y Uninformed teachers, paper found difficult, no break b/w tests for class 3

2040304301 N N Y Y N N Y Y Y Difficult format of test, shortage/excess of papers 2050304201 N N N Y N N Y Y N Papers reached late, no teacher had read the manual

2050307601 N N N Y N N N Y N Issues of time of test-easy but lengthy, papers reached late, no teacher had read the manual

2050404701 Not tested during field audit

2040301102 N N N Y N N Y Y Y Hindi paper was observed to be tough, there was no extra copy for teacher to read question for class 3

2040301701 N N N Y N N Y Y Y Hindi paper was observed to be tough, there was no extra copy for teacher to read question for class 3

2040301905 N Y N Y N N Y Y Y Hindi paper was tough, test conducted in school ground

2100501201 N Y N Y N N Y Y N No evaluator guide made available, Reading Comprehension was found to be tough

2100501202 Y Y N N N N Y Y N No evaluator guide, positive feedback 2100501401 N Y N Y Y N Y Y Y No evaluator guide, positive feedback 2100501302 N Y N Y N N Y N Y No evaluator guide, positive feedback 2100500902 N Y Y N N N N Y Y Trained teacher was not present, papers up to their level 2100501101 N N Y N N N Y N N Trained teacher was not present, issues in dealing with OMR

2100501001 N N N N Y N Y Y N Instructions not clearly followed, teacher was helping students too much

2050103601 N Y N Y N N Y Y Y 2050103801 N N N Y N N Y Y Y Teacher kept leaving class, students scared of test 2050203603 N Y N Y Y N Y Y Y


45

School Code

Did the teacher attend the evaluator training?

Were processes from the Evaluator Manual

followed?

Were there other

people present during testing

?

Did the teacher

know the

school code?

Did she help

students write

it?

Disturbance in class?

Was the Class

overcrowded?

Was the seating

arrangement APPROPRIATE for testing?

Did the teacher

have sufficient

test material?

Was time

schedule and

duration followed

?

Comments

2020601901 Y N Y Y Y N Y N Y Class 5 found paper slightly tough 2020602302 Y N Y N N N Y Y Y Wrong school code was given, teachers filled out OMR themselves

`

© Educational Initiatives Pvt. Ltd. 2013 46

At EI we believe that the real work in any assessment begins after the test is over. A deep dive into the data collected through the test is invaluable in many ways. While it obviously indicates what the students are learning well, it also gives immensely useful insights into methods of teaching and trends present in a particular school or area. For these reasons we strongly advocate the dissemination of this data at various levels.

The district report is an attempt to facilitate the efficient dissemination of the results and to provide accessible relevant reports all the stakeholders involved. Each report tries to provide statistics at the district level while providing a comparative view within the state. The contents of this report are detailed in the sections below.

5.1 A BIRD’S EYE VIEW OF THE DISTRICT’S PERFORMANCE ACROSS CLASSES AND SUBJECTS

The first section of the District wise report (Part A) includes the number of students tested in your district for each class and subject. Along with this the performance of students in your district is compared with the overall performance of all the students tested in the state.

5. A PEEK INTO THE DISTRICT REPORTS

`


5.2 SCORE DISTRIBUTION OF STUDENTS ACROSS SUBJECTS, CLASSES

Part B categorises your district’s students into buckets of performance for each class and subject. These buckets, as shared by the state, are already in use to understand how the students are distributed into buckets of higher or lower performance.

Based on this distribution the strategies for one district or class can vary a lot with respect to the other districts. For instance, in the report shown above the main aim for improving performance in Class 7 maths in this district could be to move the 67% of students (in blue), currently in the lowest buckets to the higher ones. While for Class 3 Maths, it could be to retain and expand the top 21.4% (in purple) and 22.1% (in red).

`


5.3 COMPARISON WITH OTHER DISTRICTS (COMPOSITE INDEX)

Based on the Composite Index, explained earlier in the report, Part C places each district with respect to other districts in the State. The composite index of each district is compared against the state average as well. One of the main benefits of such a comparison is to share and learn best practices from the better performing districts.

`


5.4 PERFORMANCE ON DIFFERENT SKILLS

Looking at particular skills and competencies tested can give deep insights into why the overall performance of a school / district. Part D of the district report divides all the question of the test by the comptencies tested and gives average scores of the students for each of them. These scores are also compared against the district average.

This report can be immensely useful for the teachers, text book writers to be aware of the strong and weak areas among the students.

`


5.5 QUESTION WISE PERFORMANCE ACROSS CLASSES AND SUBJECTS

Part E of the report gets one level deeper from the skills to individual items which were high or low performing. A look at these questions forces the teachers and educationists to think about why students would find a particular question too easy or difficult and therefore what does it reflect about their understanding of the concepts. Improvement areas can be easily identified through instances of very high and low performance.

`


5.6 SUBJECT LEVEL RECOMMENDATIONS

Part F summarises some recommendations at a skill level for each subject, class and district. In the report, recommendations for each skill tested for all subjects in diagnostic assessment are provided. These recommendations can be used to strengthen weak skills and maintain strong skills. These may also serve as fresh ideas to look at a concept/ skill for the teachers and educationists.

`


APPENDIX A: COMPARISON WITH ASSET

Class HP Schools Indian Private Schools Diff Stat.

Sig.? Cohen's d

Effect size N AVG SD N AVG SD

H3 8231 48.6 31.5 6564 50.3 9.5 Yes -0.1 -

M3 8231 51.3 28.6 18059 56.8 10.8 Yes -0.3 Small

H5 9151 32.1 20.1 23971 41.0 13.4 Yes -0.5 Medium

M5 9151 38.8 23.1 26330 45.6 13.4 Yes -0.4 Small

H7 9516 43.9 19.1 4245 57.2 16.2 Yes -0.8 Large

M7 9516 34.3 21.7 26201 53.3 8.9 Yes -1.2 Large

APPENDIX B: DIFFERENCE IN PERFORMANCE IN STRAIGHT FORWARD (SF) AND HIGHER ORDER (HO) QUESTIONS

Class / Subject Average Perf. SF % HO % Gap (% pt.) 3L 59.5 62.8 56.2 6.52 3M 59.7 66.7 52.6 14.13 5L 59.0 66.1 51.2 14.84 5M 48.2 52.6 43.9 8.75 7L 48.5 54.0 42.9 11.08 7M 38.1 40.6 36.0 4.60

6. APPENDIX

`


APPENDIX C: DISTRICT WISE PERFORMANCE

Language

Districts Language 3 Language 5 Language 7

N AVG SD N AVG SD N AVG SD BILASPUR 611 56.6 24.0 711 55.8 20.9 905 44.0 16.5 CHAMBA 984 53.6 21.2 951 55.2 18.1 983 46.6 14.8 HAMIRPUR 832 67.9 19.1 925 67.4 17.8 954 45.7 14.7 KANGRA 740 63.3 19.5 895 58.8 17.0 904 51.5 15.9 KULLU 746 62.4 19.5 826 60.3 16.0 972 50.3 17.2 MANDI 850 61.0 20.0 999 59.9 18.4 896 49.7 13.4 SHIMLA 864 60.0 21.5 936 58.5 17.2 809 52.7 14.2 SIRMAUR 878 52.9 21.9 882 54 18.3 946 46.3 15.3 SOLAN 650 62.1 19.8 849 59.7 17.1 777 51.2 15.1 UNA 819 60.1 21.8 893 62.7 19.1 1002 47.7 15.5 KINNAUR 90 47.2 17.6 120 47.8 16.2 110 45.5 12.2 LAHUL & SPITI 149 54.1 19.7 143 53.7 15.0 210 48.5 15.6

Maths

Districts Maths 3 Maths 5 Maths 7

N AVG SD N AVG SD N AVG SD BILASPUR 611 55 26.8 711 43.6 21.4 905 32 17.4 CHAMBA 984 56.3 26 951 44.2 21.2 983 38.5 19 HAMIRPUR 832 67.0 21.7 925 58.4 21.5 954 35.4 15.5 KANGRA 740 62.8 23.2 895 49.8 18.0 904 40.3 17.4 KULLU 746 62.7 21.4 826 48.7 17.4 972 38.9 19.0 MANDI 850 61.1 23.3 999 50.0 21.1 896 44.9 18.8 SHIMLA 864 58.1 23.1 936 46.9 19.9 809 40.4 19.2 SIRMAUR 878 54.3 25.8 882 43.9 19.2 946 33.9 17.8 SOLAN 650 60.6 22.7 849 48.4 20.2 777 38.9 17.1 UNA 819 62.1 25 893 50.2 21.5 1002 37 17.9 KINNAUR 90 49.7 20.1 120 37.6 17.5 110 29.4 10.2 LAHUL & SPITI 149 53.5 24.2 143 41.2 18.2 210 42.6 22.1

`


APPENDIX D: DISTRICT WISE COMPOSITE PERFORMANCE

All Subject Classes

District N AVG SD Cohen's d HAMIRPUR 2711 57.0 18.4 0.2 MANDI 2745 54.4 19.2 0.1 KANGRA 2539 54.4 18.5 0.1 KULLU 2544 53.9 18.4 0.1 SOLAN 2276 53.5 18.7 0.1 UNA 2714 53.3 20.1 0.1 SHIMLA 2609 52.8 19.2 0.0 STATE AVERAGE 25989 52.2 19.8 CHAMBA 2918 49.1 20.1 -0.2 BILASPUR 2227 47.8 21.2 -0.2 SIRMAUR 2706 47.6 19.7 -0.2

Language

District N AVG SD Cohen's d HAMIRPUR 2711 60.3 17.2 0.2 KANGRA 2539 57.9 17.5 0.1 KULLU 2544 57.7 17.6 0.1 SOLAN 2276 57.7 17.3 0.1 SHIMLA 2609 57.1 17.6 0.1 MANDI 2745 56.9 17.3 0.1 UNA 2714 56.8 18.8 0.1 STATE AVERAGE 25989 55.8 18.4 0.0 BILASPUR 2227 52.1 20.5 -0.2 CHAMBA 2918 51.8 18.0 -0.2 SIRMAUR 2706 51.1 18.5 -0.3

Maths District N AVG SD Cohen's d

HAMIRPUR 2711 53.6 19.6 0.2 MANDI 2745 52.0 21.1 0.2 KANGRA 2539 51.0 19.5 0.1 KULLU 2544 50.1 19.3 0.1 UNA 2714 49.8 21.5 0.0 SOLAN 2276 49.3 20.0 0.0 STATE AVERAGE 25989 48.8 21.1 0.0 SHIMLA 2609 48.5 20.7 0.0 CHAMBA 2918 46.3 22.1 -0.1 SIRMAUR 2706 44.0 20.9 -0.2 BILASPUR 2227 43.5 21.9 -0.3

`
