Methods for Assessing Competences in Higher · PDF fileMethods for Assessing Competences in Higher Education ... performance based training ... Advantages . Disadvantages

Methods for Assessing Competences in Higher Education | PD Dr. Edith Braun |

2016 The Higher Education Conference

Methods for Assessing Competences in Higher Education

A Comparative Overview

Edith Braun (INCHER-Kassel)

1

20.07.2016



20.07.2016

2

• Social changes and political background • Competence-based Teaching and Quality Assurance

– Definition of competences

• Assessment of competences – 5 different approaches

• Discussion

Structure



Need for competences assessment

Social changes: demographic change, knowledge society, expansion of higher education systems, life long learning Political reaction: Bologna Declaration (1999); Lisbon Strategy (2000); economic crisis of 2008 Quality assurance focus on learning outcomes, more specific on competences

20.07.2016

3



20.07.2016

4

1. Aims of teaching and learning in HE 1. Scientific knowledge 2. Personal development 3. Preparation for the labour market

• Competence based, performance based training (Schaper, Wildt, Reis, Horvath & Bender, 2012)

• Employability (De Vos, De Hauw, & Van der Heijden, 2011) 2. Quality assurance in HE

• Accreditation • Framework of Qualifications

Need for competences assessment



Definition of Masten and Coatsworth (1998)

‘a pattern of effective adaptation to the environment, […] broadly defined in terms of reasonable success with major developmental tasks expected for a person of a given age and gender in the context of his or her culture, society, and time […] This definition takes into account the dual meaning of competences and considers track record of achievements in the past (competent performance) as well as individual’s capability to perform well in future.’

"competences“

• Goes beyond cognitive skills • Enable to master complex situation • Adapt to new situations as well as become effective citizens • Cognitive (discipline-specific) and non-cognitive aspects of competences: personal

and social skills, leadership and communication skills

How can we assess these competences?

5

Definition



1. Self-report of competences

2. Job requirement approach

3. Student engagement approach

4. Achievement tests

5. Performance based testing Role plays (own research)

Five established approaches:

20.07.2016

6

Assessment approaches



20.07.2016

7

Self-report of competences rely on students’ self-perceptions and assessment of their competences. CIRP (Sax, Astin, Korn, & Mahoney, 2000): • “rate your current level of competence in comparison to an average

person of your age“ • ‘‘Highest 10%’’, ‘‘Above average’’, ‘‘Average’’, ‘‘Below average’’ and

‘‘Lowest 10%’’ • Questionnaire:

– „Leadership ability“ – „Public speaking ability“ – „Self-confidence (social)“




20.07.2016

8

Validation Correlations “self-rated skills” and grades:

– CEQ: r= .23 (Wilson, Lizzio, & Ramsden, 1997) – SIR II: r= .20 (Centra, 2010)

Observed opportunities to learn and self-reports of competences (Braun & Hannover 2011): Factual knowledge: γ01=.31 Presentation skills γ01=.32 Personal skills γ01=.52




20.07.2016

9

Advantages

• Help to assess how students perceive themselves

• Meaningful correlations between self-reports and behavior

• External validity of self-reports Systematic correlations with external

criteria (independent observer ratings, course grades, learning styles of students)

• Assessment of communication skills and personal/social development through items on the questionnaires Items are found to be internally

consistent with each other (high Cronbach’s alpha)

• Simple and cost-efficient

Disadvantages

• Not completely free of biases

Questionable quality of self-assessment, in case of vague formulation of items

Prone to cultural and context-specific influences and interpretations




Basic assumptions:

1. Activities that individuals engage in at their workplace reflect the demands and requirements of their work.

2. The best way to get information about job-related activities and requirements is to ask the employees directly.

3. Generally, employees are able to share information about their duties and requirements at work in a less biased manner and also indicate how often they engage in what type of activities.

4. Compared to self-report JRA allows for using job-related activities and requirements as a potentially less biased proxy for measuring job-related competences.

Approach for measuring vocational tasks and activities

20.07.2016

10

2. Job requirement approach



Job requirement approach in a German-wide survey of HE graduates • Included over 10,000 graduates spanning across different disciplines and higher

education institutions • Participants were asked directly about their job-related activities and requirements • 41 items

Area of competences Item Example Numbers of Items

Planning and organising of work processes

At work, I schedule processes for others. 8

Promoting others At work, I have to instruct and lead a group of persons in a structured way.

3

Leading At work, I advise others. 5 Dealing autonomously with challenging tasks

At work, I have to consider possible implications for other areas or persons.

9

Information processing At work, I document complex issues. 6 Number processing At work, I plan prices, costs or budgets. 3 Communicating in foreign languages

At work, I maintain contact with persons from other cultures.

4

Physical performance At work, I operate technical equipment, machinery or systems. The operation of PCs is not included at this point.

3

Each item was answered on a completely labelled and numbered five-point Likert scale: (1) never; (2) less than once a month; (3) at least once a month but less often than once a week; (4) at least once a week but not daily; (5) daily

20.07.2016

11



20.07.2016

12

• Based on self-descriptions and cannot differentiate between various levels of competences

I suggest a cautious interpretation of activities as a proxy for competences

• Helpful in identifying activities that higher education graduates engage in at work

• Identification of share of graduates who are on an (in-)adequately position

• Addresses many of the weakness of self-ratings

• Takes into account the actual activities and behaviors that a person is involved with

• Questions are asked within the context of the work environment

2. Job requirement approach Advantages Disadvantages



This approach has been employed in surveys for assessing students’ learning environment and activities they engage in at higher education institutions . (Kuh, 2003) “The quality of effort students themselves devote to educationally purposeful activities that contribute directly to desired outcomes.” (Hu & Kuh, 2002)

Four most prominent subscales (‘Benchmarks of Effective Educational Practices’; McCormick, A. C. , J. Kinzie & R. M. Gonyea; 2013):

1. academic challenge 2. active and collaborative learning 3. student-faculty interaction 4. supportive campus environment

20.07.2016

13




20.07.2016

14

• Unclear definition of ‘educationally purposeful activities’ need for more theoretically and

empirically driven research • Empirical correlations between the

surveyed activities and more objectives measurements of learning outcomes (GPA) are quite weak

• Helps to identify aspects of teaching and learning which need improvement

• Students provide relevant information regarding activities they have been involved with in higher education

• Adoption of the NSSE survey in itself is an intervention in promoting students’ engagement

• Opportunity to voice the opinion on the overall learning process and environment which could in turn lead to better engagement in higher education

• Main advantage is its underlying emphasis on learning opportunity

Advantages Disadvantages




Direct measurement of the level of competences of students and graduates • Measuring cognitive, mostly discipline-specific competences or

knowledge A lack of tests available for objectively assessing students’

competences in higher education

It is important to bear in mind the distinction between competences and intelligence.

20.07.2016

15




Inter-institutional studies based on direct assessment of competences in higher education; large scale implementation

AHELO (Assessment of Higher Education Learning Outcomes)

International and inter-institutional comparison of competences in higher education system

• Implemented on a pilot basis between 2009 and 2012 • 17 countries participated • Areas of competences: ‘generic skills strand’, ‘economics strand’, and

‘engineering strand’

20.07.2016

16

Example 1: AHELO



20.07.2016

17

Example 2: KoKoHs

KoKoHs (Competency models and instruments of competence assessment in higher education)

Created a major propulsion and closed gaps in the area of competence assessment in higher education

• First funding period (2011-2015) 24 projects were initiated • Second funding period (2015-2019) 15 projects have been initiated



20.07.2016

18

• Scientific and methodological

challenges

• Students are often not motivated to take achievement tests

• Do not take into account the complexity of competences

• Direct measurement of the level of competences of students and graduates

Advantages Disadvantages




Test Development for measuring communication skills for economics and educational science students in Germany

Advantages:

• Socially accepted method to activate competences, which are of interests • Sufficient method to assess competences in the way we have defined it in

the beginning • The students are highly motivated to participate

20.07.2016

19

5. Performance based testing Role plays ‘KomPrü’



Test Development for measuring communication skills for economics and educational science students in Germany • 32 role plays • Based on Habermas’ theory (1984) of communicative action • piloted at the University of Kassel in Germany, Feedback of 6 experts • Survey phase in Fall 2016 11 higher education institutions in Germany 500 economics and educational science students

Allow for assessing competences in an objective way on an individual level but more research is needed to prove its transferability for assessing competences in other disciplines

20.07.2016

20




20.07.2016

21

• Prospective teacher has to admonish a difficult pupil

• [video]




20.07.2016

22

Item

The testperson raises clearly that he / she does not accept the behavior of the interlocutor. O O O O

The testperson makes clear that he / she does not want to hear any explanations / excuses. O O O O

The testperson remains on-topic (unacceptable behavior of the interlocutor). O O O O

The testperson reasons his / her point of view consistently. O O O O

Observation form

Allow for competence assessment in an objective way on an individual level but more research is needed to prove its transferability for assessing competences in other disciplines


CONCLUSION

20.07.2016 23



20.07.2016

24

Self-reports

Job requirements

Student engagement

Achievement tests

Role plays

Definition of competence

Mostly not existing

Self-rated performance of job-related activities

learning opportunities and activities that students are engaged in

Cognitive skills and/or knowledge

Performance in complex and authentic situations

Indirect/ direct assessment

Indirect assessment

Indirect assessment

Indirect assessment

Direct assessment Direct assessment

Added value (changes in competence over time)

No Less quantifiable

Less quantifiable Quantifiable on level of competences

Quantifiable on level of competences

Effort of development

Required effort often under-estimated

Often well developed




Effort of applying Low Low Low High High

Students` Motivation to participate

Medium Medium Medium Low High



It is essential to either use these approaches in combination or to develop innovative methods which are equally suitable for measuring discipline-related as well as more generic competences such as creativity, social well-being

20.07.2016

25



20.07.2016

26

Subsumption

Macro level: Local & regional context

Meso level: Course of studies

Micro level: Courses

Individual level: Students / lecturer

Structure Process Result

Braun, Weiß & Seidel (2014)

Stru

ctur

Supp

ort

Orie

ntat

ion

Chal

leng

e Exo level: University as an institution



20.07.2016

27



References • Alkin, M. C. & Christie, C. A. (2002). The Use of Role-Play in Teaching Evaluation. American Journal of Evaluation, 23(2), 209–

218. • Allen, J., Ramaekers, G., Velden, R. van der., (2005) Measuring Competencies of Higher Education Graduates. In D. J. Weerts &

J. Vidal (Eds.), Enhancing Alumni Research: European and American Perspectives, New Directions for Institutional Research. Number 126 (pp. 49-59). Jossey-Bass, San Francisco.

• Astin, A. (1999). Student involvement: A developmental theory for higher education. Journal of College Student Development, 40(5), 518-529.

• Beard, R. L., Salas, E., & Prince, C. (1995). Enhancing transfer of training: using role-play to foster teamwork in the cockpit. The International journal of aviation psychology, 5(2), 131–143.

• Blömeke, S., Gustafsson, J. E., & Shavelson, R. J. (2015). Beyond Dichotomies. Zeitschrift für Psychologie, 223 (1), 3-13. • Blömeke, S., Zlatkin-Troitschanskaia, O., Kuhn, C., & Fege, J. (2013). Modeling and measuring competencies in higher

education: Tasks and challenges. In Blömeke, S., Zlatkin-Troitschanskaia, O., Kuhn, C., & Fege, J. (eds.) Modeling and measuring competencies in higher education: Tasks and challenges, Rotterdam, The Netherlands: Sense Publishers.

• BMBF (Bundesministeriums für Bildung und Forschung), Koordination DZHW (2013). Datenportal. http://www.datenportal.bmbf.de/portal/de/K25.gus.

• Bologna Declaration. (1999). The Bologna declaration of June 1999. Joint declaration of the European Ministers of Education. Retrieved from http://www.ond.vlaanderen.be/hogeronderwijs/bologna/documents/ MDC/BOLOGNA_DECLARATION1.pdf.

• Boni, A., & Lozano, J.F. (2007). The generic competences: An opportunity for ethical learning in the European convergence in higher education. Journal of Higher Education, 54, 819-883.

• Braun, E. M., & Brachem, J. C. (2015). Requirements Higher Education Graduates Meet on the Labor Market. Peabody Journal of Education, 90(4), 574-595.

• Braun, E., & Leidner, B. (2009). Academic course evaluation: Theoretical and empirical distinctions between self-rated gain in competences and satisfaction with teaching behavior. European Psychologist, 14(4), 297-306.

• Braun, E., Athanassiou, G., Pollerhof, K. & Schwabe, U. (submitted). Wie lassen sich kommunikative Kompetenzen messen? Konzeption einer kompetenzorientierten Prüfung kommunikativer Fähigkeiten.

20.07.2016

28



20.07.2016

29

References • Braun, Edith, Alan Woodley, John TE Richardson, and Bernhard Leidner. "Self-rated competences questionnaires from a design

perspective." Educational Research Review 7, no. 1 (2012): 1-18. • Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages*. Research in

higher education, 47(1), 1-32. • Centra, J. (1998). The development of the Student Instructional Report II. Princeton, NJ: Educational Testing Service • Chan, R.Y., Brown, G.T.L., & Ludlow, L.H. (2014). What is the purpose of higher education?: A comparison of institutional and

student perspectives on the goals and purposes of completing a bachelor’s degree in the 21st century. Paper to be presented at the annual American Education Research Association (AERA) conference. Philadelphia, PA: April 5, 2014. Retrieved from https://www.dal.ca/content/dam/dalhousie/pdf/clt/Events/Chan_Brown_Ludlow%282014%29.pdf

• Chapman, J.D. and Aspin, D.N. (2013). A problem-solving approach to addressing current global challenges in education. British Journal of Educational Studies, 61(1), 49-62.

• Coates, H., & Mahat, M. (2014). Assessing student engagement and outcomes: Modelling insights from Australia and around the world. International Journal of Chinese Education, 2(2), 241-264.

• Dublin Descriptors. (2004). Shared Dublin Descriptors for Bachelor’s, Master’s and Doctoral awards. Retrieved from http://www.tempus.ac.rs/here/tl_files/Dokumenti/Dublinski%20deskriptori.pdf.

• Enders, J., De Boer, H., File, J., Jongbloed, B., & Westerheijden, D. (2011). Reform of higher education in Europe (pp. 1-10). SensePublishers.

• European Commission. (2011). Supporting growth and jobs- an agenda for the modernisation of Europe’s higher education systems. Brussels: European Commission.

• Felstead, A. et al., 2007. Skills At Work, 1986 to 2006, Oxford: ESRC Centre on Skills, Knowledge and Organisational Performance.

• Gonyea, R. M., Kish, K. A., Kuh, G. D., Muthiah, R. N., & Thomas, A. D. (2003). College Student Experiences Questionnaire. Norms for the fourth edition. Bloomington, IN: Indiana University Center for Postsecondary Research, Policy and Planning.

• Gordon, J., Ludlum, J., & Hoey, J. J. (2008). Validating NSSE against student outcomes: Are they related?. Research in Higher Education, 49(1), 19-39.

https://www.dal.ca/content/dam/dalhousie/pdf/clt/Events/Chan_Brown_Ludlow(2014).pdf



20.07.2016

30

References • Guliker, J. T. M., Bastiaens, T. J., Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educational

Technology Research and Development, 52(3), 67-86. • Habermas, J. (1984). The theory of communicative action, Volume I. Boston: Beacon. • Harden, R. M., & Gleeson, F. A. (1979). Assessment of clinical competence using an objective structured clinical examination

(OSCE). Medical education, 13(1), 39-54. • Hartig, J., Klieme, E., & Leutner, D. (Eds.). (2008). Assessment of competencies in educational contexts. Hogrefe Publishing. • Hovdhaugen, E. (2013). AHELO: Norway’s Experience of Participation in the Feasibility Study. In Braun, E., Donk, A., & Bülow-

Schramm, M. (Eds.). AHELO goes Germany. Hannover: HIS. 7-12. • Klaukien, A., Ackermann, D., Helmschrott, S., Rammstedt, B., Solga, H. & Wößmann, L. (2013). Grundlegende Kompetenzen

auf dem Arbeitsmarkt. In B. Rammstedt (Ed.), Grundlegende Kompetenzen Erwachsener im internationalen Vergleich. Ergebnisse von PIAAC 2012 (p. 127-166). Münster: Waxmann.

• Koucky, Jan, Christoph Meng, and Rolf Van Der Velden. "Reflex country study." Maastricht: Research Centre for Education and the Labour Market/REFLEX (2007).

• Kuh, G. D. (2003). What we're learning about student engagement from NSSE: Benchmarks for effective educational practices. Change: The Magazine of Higher Learning, 35(2), 24-32.

• Kuh, G. D. (2009). The National Survey of Student Engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 141, 5–20.

• Kunina, O., Wilhelm, O., Formazin, M., Jonkmann, K., & Schroeders, U. (2007). Extended criteria and predictors in college admission: Exploring the structure of study success and investigating the validity of domain knowledge. Psychology Science, 49(2), 88.

• Kyllonen, P. (2005). The case for noncognitive assessments. R & D Connections, 1-7. • Lawless, C., & Richardson, J. T. E. (2004). Monitoring the experiences of graduates in distance education. Studies in Higher

Education, 29, 353–374. • Lisbon Strategy. (2000). The Lisbon Strategy in short. Retrieved from

https://portal.cor.europa.eu/europe2020/Profiles/Pages/TheLisbonStrategyinshort.aspx.

https://portal.cor.europa.eu/europe2020/Profiles/Pages/TheLisbonStrategyinshort.aspx



20.07.2016

31

References • Lozano, J. F., Boni, A., Peris, J., & Hueso, A. (2012). Competencies in higher education: A critical analysis from the capabilities

approach. Journal of Philosophy of Education, 46(1), 132-147. • Lucas, R., & Baird, B. (2006). Global self assessment. In M. Eid & E. Diener (Eds.), Handbook of psychological measurement: A

multimethod perspective (pp. 29–42). Washington, DC: American Psychological Association. • Masten, A.S. & Coatsworth, J.D. (1998). The development of competence in favorable and unfavorable environments.

American Psychologist 53 (2), 205-220. • Mulder, M., Gulikers, J., Wesselink, R., & Biemans, H. (2008). The new competence concept in higher education: error or

enrichment. Paper presented to AERA New York. Retrieved from http://www.mmulder.nl/PDF%20files/2008-03-31%20Paper%20Mulder%20AERA%202008.pdf.

• OECD (2013). OECD Skills Outlook 2013. First results from the survey of adult skills. Paris: OECD. • Pascarella, E. T., Seifert, T. A., & Blaich, C. (2010). How effective are the NSSE benchmarks in predicting important educational

outcomes?. Change: The Magazine of Higher Learning, 42(1), 16-22. • Peterson, N. G., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K. Y., Campion, M. A., Mayfield, M. S., Morgeson, F. P.,

Pearlman, K., Gowing, M. K., Silver, M. B. & Dye, D. M. (2001). Understanding work using the Occupational Information Network (O*NET): Implications for practice and research. Personnel Psychology, 54(2), 451-492.

• Sax, L. J., Astin, A. W., Korn, W. S., & Mahoney, K. M. (1999). The American Freshman: National Norms for Fall 1999. Higher Education Research Institute, UCLA Graduate School of Education and Information Studies, 3005 Moore Hall, Mailbox 951521, Los Angeles, CA 90095-1521.

• Spinath, B., Spinath, F. M., Harlaar, N., & Plomin, R. (2006). Predicting school achievement from general cognitive ability, self-perceived ability, and intrinsic value. Intelligence, 34(4), 363-374.

• Umbach, P. D., & Wawrzynski, M. R. (2005). Faculty do matter: The role of college faculty in student learning and engagement. Research in Higher Education, 46(2), 153-184.

• Van Ments, M. (1999). The effective use of role-play: Practical techniques for improving learning (2nd ed.). London: Kogan Page.

http://www.mmulder.nl/PDF files/2008-03-31 Paper Mulder AERA 2008.pdf

http://www.mmulder.nl/PDF files/2008-03-31 Paper Mulder AERA 2008.pdf



20.07.2016

32

References • Walker, M. (2003). Framing social justice in education: What does the capabilities approach offer? British journal of

educational studies, 51, 168-187. • Westerheijden, D. F., Stensaker, B., & Rosa, M. J. (Eds.). (2007). Quality assurance in higher education: Trends in regulation,

translation and transformation. Dordrecht, NL: Springer. • Wilson, K., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of the Course Experience

Questionnaire. Studies in Higher Education, 22, 33–53. • Winteler, A. & P. Forster (2008). Lern-Engagement der Studierenden: Indikator für die Qualität und Effektivität von Lehre und

Studium. Das Hochschulwesen, 56, 162-170. • Zlatkin-Troitschanskaia, O., Pant, H. A., Kuhn, C., Toepper, M. & Lautenbach, C. (2016). Messung akademisch vermittelter

Kompetenzen von Studierenden und Hochschulabsolventen. Ein Überblick zum nationalen und internationalen Forschungsstand. Wiesbaden: Springer.

• Zlatkin-Troitschanskaia, O., Shavelson, R.J., & Kuhn, C. (2015). The international state of research on measurement of competency in higher education. Studies in higher education, 40(3), 393-411.



Edith Braun INCHER-Kassel

Universität Kassel [email protected]

Remarks?

Questions?

20.07.2016

33

Comments and Discussion

Documents

Methods for Assessing Competences in Higher · PDF fileMethods for Assessing Competences in Higher Education ... performance based training ... Advantages . Disadvantages