An Investigation of Assessment An Investigation of Assessment Policy and Practice in New Policy and Practice in New Zealand Higher Education Zealand Higher Education

InstitutionsInstitutions

Susan DavidsonSusan Davidson11, Helen Anderson, Helen Anderson22, Malcolm Rees, Malcolm Rees33, Lynanne , Lynanne McKenzieMcKenzie11, Patricia M. Johnston, Patricia M. Johnston44, ,

Richard B. FletcherRichard B. Fletcher55, Luanna H. Meyer, Luanna H. Meyer11

11Jessie Hetherington Centre for Educational Research, Victoria UnJessie Hetherington Centre for Educational Research, Victoria University, Wellington, NZiversity, Wellington, NZ22Manukau Institute of Technology, Manukau, NZManukau Institute of Technology, Manukau, NZ33Massey University, Palmerston North, NZMassey University, Palmerston North, NZ44Te Whare Te Whare WWāānangananga o o AwanuiAwanuiāārangirangi, , WhakataneWhakatane, NZ, NZ55Massey University, Albany, NZMassey University, Albany, NZ

BackgroundBackgroundThe research is part of a 3 year project to investigate The research is part of a 3 year project to investigate Valid and Practical Tertiary Assessment of Student Valid and Practical Tertiary Assessment of Student Learning Outcomes, Learning Outcomes, 20062006--2008, supported by the 2008, supported by the Teaching & Learning Research Initiative (TLRI) fund Teaching & Learning Research Initiative (TLRI) fund managed by the New Zealand Council for Educational managed by the New Zealand Council for Educational Research (NZCER)Research (NZCER)

Our research project is a collaboration across four Our research project is a collaboration across four public tertiary institutions (TEIs) including public tertiary institutions (TEIs) including Victoria Victoria UniversityUniversity (Professor Luanna Meyer, Project Director); (Professor Luanna Meyer, Project Director); Massey UniversityMassey University (Dr Richard Fletcher & Malcolm (Dr Richard Fletcher & Malcolm Rees, CoRees, Co--Directors); Directors); Te Whare Te Whare WWāānangananga o o AwanuiAwanuiāārangirangi (Professor Trish Johnston, Co(Professor Trish Johnston, Co--Director); Director); and and Manukau Institute of TechnologyManukau Institute of Technology (Dr Helen (Dr Helen Anderson, CoAnderson, Co--Director)Director)

Data Sources for the ResearchData Sources for the Research

1.1. Staff and student surveys about attitudes Staff and student surveys about attitudes towards and uses of assessment towards and uses of assessment

2.2. Documents analysisDocuments analysis——institutional policy institutional policy and practiceand practice

3.3. Interviews with key policy leaders and Interviews with key policy leaders and decisiondecision--makersmakers——at institutional and at institutional and faculty levelsfaculty levels

Survey Data from Academic StaffSurvey Data from Academic Staff

Staff survey about assessmentStaff survey about assessmentFullFull--time and parttime and part--time academic staff from 4 TEIs time academic staff from 4 TEIs (N=879) across disciplines(N=879) across disciplines2 universities, 1 polytechnic, 1 2 universities, 1 polytechnic, 1 wwāānanganangaLikertLikert--ratings of questions about assessment policy, ratings of questions about assessment policy, purpose and practicespurpose and practicesResponses to openResponses to open--ended question on the survey ended question on the survey ““Are there any other issues about assessment Are there any other issues about assessment practices you would like to share with us?practices you would like to share with us?”” (N=159)(N=159)

Documents AnalysisDocuments Analysis

Focused on comprehensive review of all Focused on comprehensive review of all documents related to assessment policy documents related to assessment policy and practice from the 4 TEIs in Study 1 (2 and practice from the 4 TEIs in Study 1 (2 universities, 1 polytechnic, 1 universities, 1 polytechnic, 1 wwāānangananga ))

Documents sourced from key staff at Documents sourced from key staff at central, faculty and department levels and central, faculty and department levels and comprised both print and web publicationscomprised both print and web publications

Interview DataInterview DataData SourcesData Sources::

2 key academic managers from each of an expanded 2 key academic managers from each of an expanded group of seven TEIs (4 project TEIs plus 3 additional), group of seven TEIs (4 project TEIs plus 3 additional), including in total 3 universities, 2 polytechnics and 2 including in total 3 universities, 2 polytechnics and 2 wwāānangananga (interviewee N=14)(interviewee N=14)Interview content informed by results from the survey Interview content informed by results from the survey and documents analysesand documents analysesQuestion focus: aspects of managing existing policy; Question focus: aspects of managing existing policy; processes for policy & practice development; processes for policy & practice development; professional development; national context; grading professional development; national context; grading practices; student outcomes; feedback policy & practicespractices; student outcomes; feedback policy & practices

FindingsFindings

Multiple purposes of assessmentMultiple purposes of assessmentAssessment Assessment ofof versus versus for for learninglearning

Consequences of focus on proceduresConsequences of focus on proceduresOther important considerationsOther important considerations

StudentsStudentsCultural appropriateness/responsivenessCultural appropriateness/responsivenessESOL and other language issuesESOL and other language issuesDisability adaptationsDisability adaptationsShifts in assessment policy and practicesShifts in assessment policy and practices

1 1 http://http://www.standards.dfes.gov.uk/www.standards.dfes.gov.uk/

personalisedlearning/five/aflpersonalisedlearning/five/afl//

Purposes of assessmentPurposes of assessment

Assessment Assessment ofof learninglearningMeasuring what/how much students have learned Measuring what/how much students have learned

(knowledge, skills, etc.) with respect to learning (knowledge, skills, etc.) with respect to learning outcomes and/or graduate profileoutcomes and/or graduate profile

Assessment Assessment forfor learninglearning“…“…using evidence and feedback to identify where pupils are using evidence and feedback to identify where pupils are

in their learning, what they need to do next and how in their learning, what they need to do next and how best to get achieve this.best to get achieve this.””11

““The primary point of classroomThe primary point of classroom--based based assessment techniques is to get the assessment techniques is to get the students to identify their existing students to identify their existing knowledge and opinions and to provide knowledge and opinions and to provide some degree of critical thinking skills. The some degree of critical thinking skills. The greatest danger we face at university greatest danger we face at university levellevel……is assessment that focuses on is assessment that focuses on REPETITION and MEMORY rather than the REPETITION and MEMORY rather than the ability to think.ability to think.””

Consequences of a focus on Consequences of a focus on proceduresprocedures

Constraints: types, time and workload, Constraints: types, time and workload, feedback to students, external regulatorsfeedback to students, external regulatorsTensions about policies and requirements: Tensions about policies and requirements: time, workload expectations, scaling, time, workload expectations, scaling, ““Dumbing downDumbing down”” courses to pass studentscourses to pass studentsLack of knowledge about pedagogically Lack of knowledge about pedagogically sound assessment practicessound assessment practices

““Types of assessment are often chosen Types of assessment are often chosen because of their outcome or applied because of their outcome or applied values values –– e.g. easy to mark (exams, tests), e.g. easy to mark (exams, tests), easy to prepare (essay questions), easy to prepare (essay questions), prevention of plagiarism, etc. prevention of plagiarism, etc. –– rather rather than their value as a learning tool.than their value as a learning tool.””

““That student grades are redistributed to That student grades are redistributed to conform to a standard distribution curve is conform to a standard distribution curve is unethical. If we mark to criteria and the unethical. If we mark to criteria and the student achieves those criteria the student achieves those criteria the university has no right to then rank those university has no right to then rank those students against each otherstudents against each other…”…”

Students

Want/expect easier material, easier assessmentsLower ability and/or less preparation for tertiary levelFeelings of entitlement because they are payingWilling to challenge teachers by taking complaints to academic managementStaff and managers’ concern for student learning, future careers

Cultural appropriateness/responsiveness

Mostly wwāānangananga participantsMaori cultural needs and viewpoints, not just in assessment but entire teaching and learning processE.g. peer teaching/assessing, “ako”concept, group learningAuthentic assessments, e.g. traditional protocols

Other issues

ESOL and other language issuesDisability adaptations

Shifts in assessmentShifts in assessmentFrom mostly/fully external to more From mostly/fully external to more internalinternalIncreasing emphasis on learning outcomes Increasing emphasis on learning outcomes in the graduate profile in the graduate profile versusversus grading on a grading on a curvecurveManaging plagiarism and cheatingManaging plagiarism and cheating

Different foci for different groupsDifferent foci for different groups

Institution typeInstitution typeWWāānangananga: focus on culturally appropriate assessment: focus on culturally appropriate assessmentExternal External vsvs internal regulation (wananga/polytechs internal regulation (wananga/polytechs vsvsuniversities)universities)Grading systems all different across TEIsGrading systems all different across TEIs

StaffStaffFocus on constraints, purposes and practices of Focus on constraints, purposes and practices of assessmentassessment

ManagersManagersFocus on regulationsFocus on regulations

DiscussionDiscussion

Consequences of focus on proceduresConsequences of focus on proceduresStudent expectationsStudent expectationsCultural appropriateness/responsivenessCultural appropriateness/responsivenessESOL and other language issuesESOL and other language issuesInternational assessment policy contextInternational assessment policy context

Consequences of a focus on Consequences of a focus on procedures and requirementsprocedures and requirements

Assessment Assessment of of learning reflects need for validity, learning reflects need for validity, reliability, utility, consistency and equity in reliability, utility, consistency and equity in grading and marking, grading and marking, but but

An exclusive or even primary focus on these issues An exclusive or even primary focus on these issues as the major TEI level drivers can constrain as the major TEI level drivers can constrain creativity in assessment practices, appropriate creativity in assessment practices, appropriate disciplinary variations in assessment, and disciplinary variations in assessment, and feedback and feedfeedback and feed--forward processes to guide forward processes to guide student learning and improve learning outcomesstudent learning and improve learning outcomes

Context: Student expectationsContext: Student expectationsCompleting qualifications in a feesCompleting qualifications in a fees--paying, student loan paying, student loan environmentenvironmentEmphasis on fairness, culture, equity and expectations Emphasis on fairness, culture, equity and expectations about success vs. Failureabout success vs. FailurePrevious assessment experiences: School leavers enter Previous assessment experiences: School leavers enter NZ TEIs with the NCEA that has exposed them to a mix NZ TEIs with the NCEA that has exposed them to a mix of internal and external assessment; feedback/feedof internal and external assessment; feedback/feed--forward on their work; opportunities to reforward on their work; opportunities to re--sit and sit and resubmit; choices in assessment participation. resubmit; choices in assessment participation. More More sophisticated, practiced and accountabilitysophisticated, practiced and accountability--focussed than focussed than the previous the previous ““bursarybursary”” generationgeneration

Culturally appropriate and culturally Culturally appropriate and culturally responsive assessmentresponsive assessment

Ensuring that assessment is culturally responsive & Ensuring that assessment is culturally responsive & appropriate does not entail different outcome appropriate does not entail different outcome criteria, but instead requires quality of outcomecriteria, but instead requires quality of outcome

Assessment context and processes (including Assessment context and processes (including underlying policy & practices) should, when underlying policy & practices) should, when appropriate, be varied to reflect cultural values appropriate, be varied to reflect cultural values and practiceand practice

Examples: Examples: Inclusion of group assessment tasks with Inclusion of group assessment tasks with collective participation and shared marks/grades; oral as collective participation and shared marks/grades; oral as well as written assignments; storytelling assessments, well as written assignments; storytelling assessments, and so on and so on

ESOL and other language issuesESOL and other language issuesExamination of practices that advantage and disadvantage Examination of practices that advantage and disadvantage

students by assessment that are not solely influenced by students by assessment that are not solely influenced by mastery of learning outcomes mastery of learning outcomes but also (heavily) but also (heavily) influenced by influenced by linguistic abilitylinguistic ability

Issues:Issues:When/how does the graduate profile and/or professional When/how does the graduate profile and/or professional registration require English language proficiency as well registration require English language proficiency as well as disciplinary mastery?as disciplinary mastery?When/how do particular assessments unfairly When/how do particular assessments unfairly disadvantage students who are processing multiple disadvantage students who are processing multiple language proficiencies? (for example, timed language proficiencies? (for example, timed examinations and/or timeexaminations and/or time--constrained essays)constrained essays)

International assessment International assessment policy contextpolicy context

Institutional accountability: prove what Institutional accountability: prove what and how much students are learning and how much students are learning Institutional comparability on student Institutional comparability on student learninglearningProblem of academic staffsProblem of academic staffs’’ lack of lack of assessment trainingassessment trainingAssessment Assessment forfor versus versus ofof learninglearning

SummarySummary

Tertiary sector assessment policies: where Tertiary sector assessment policies: where to from here?to from here?