View
23
Download
1
Category
Preview:
DESCRIPTION
Institut für Forschungsinformation und Qualitätssicherung. Graduate surveys as an outcome evaluation Presentation EAIR Forum 2009, July 23-26, Vilnius, Lithuania Dr. René Krempkow Institute for Research Information and Quality Assurance, Bonn (Germany) - PowerPoint PPT Presentation
Citation preview
Institut für Forschungsinformation und Qualitätssicherung
Graduate surveys as an outcome evaluation
Presentation EAIR Forum 2009, July 23-26, Vilnius, Lithuania
Dr. René Krempkow Institute for Research Information and Quality Assurance, Bonn (Germany)
(former project leader of Freiburg Graduate Surveys, University of Freiburg)
Institut für Forschungsinformation und Qualitätssicherung
Institut für Forschungsinformation und Qualitätssicherung
iFQ
OutlineOutline
1. Relevance of graduate surveys for Quality Management
2. Measuring outcomes of higher education – Influencing factors of professional success
3. Exemplary results
4. Interpretation of results
5. Prospects for analysis
Institut für Forschungsinformation und Qualitätssicherung
Graduate Surveys
Institut für Forschungsinformation und Qualitätssicherung
iFQ
When integrating graduate surveys in quality management:
1. Adequate interpretation of results only through comparison with other HEIs and more general national data (see Teichler 2003)
2. Account for individual characteristics through institution-specific and programme-specific surveys (see QM-literatur)
Nationwide “Core Questionnaire” with institution-/programme-specific adaptations
The Potential of graduate surveys The Potential of graduate surveys for Quality Management Ifor Quality Management I
Institut für Forschungsinformation und Qualitätssicherung
iFQ
• Analyse data to identify determinants of professional success to feed back to Quality Management system
• Unit of observation: study programmes
• Account for specific context by interpreting data taking into account influencing factors (e.g. regional economy)
• Account for specific initial conditions, e.g. % of student parents, socioeconomic background of students (Schomburg/Teichler, 1998)
The Potential of graduate surveys The Potential of graduate surveys for Quality Management II for Quality Management II
Institut für Forschungsinformation und Qualitätssicherung
Source: Schomburg/ Teichler (1998: 165)
• Two HEI stand out positively• To what degree can differences be attributed to different starting
conditions of HEIs? This will be discussed on the basis of Freiburg Graduate Surveys and Dresden Graduate Surveys.
The Potential of graduate surveys The Potential of graduate surveys for Quality Management III for Quality Management III
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Measuring Outcome with graduate Measuring Outcome with graduate surveys – what does professionell surveys – what does professionell success depend on?success depend on?
• Multivariate analysis allows identifying factors influencing professionell success
• This information is independant from (subjective) self-assessment of graduates (see Krempkow/Pastohr 2006)
• Example: job experience, study times abroad and specific competences appear to be more important than study duration and grades – That leads to the question: Can and (if yes) What can HEIs do to improve professional success of graduate?
• Multivariate (correlatory) analysis allows analysing relationships between professional success and aspects of study, including starting conditions and context!
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Influencing factors on professional Influencing factors on professional successsuccessObjective and subjective criteria of professional success as indicators
of HE outcomes (Teichler/Schomburg (1997: 248)
1. Objective indicators for professional success (e.g. income, hierarchical position)
2. Subjective indicators for professional success (e.g. job satisfaction, autonomy)
3. Objective indicators for transition HE-labour market (e.g. Time of job hunt)
4. Subjective assessment of goodness-of-fit HE – job (e.g. usefulness of study contents, employment adequacy).
• Freiburg and Dresden Survey: Gross income, job satisfaction and employment adequacy fitting criteria for professional success.
Institut für Forschungsinformation und Qualitätssicherung
Basic model of influencing factors on professional Basic model of influencing factors on professional successsuccess
INDIVIDUAL CHARACTERIS-TICS + SOCIO-ECONOMIC BACKGROUND
CHARACTERISTCS OF HIGHER EDUCATION (PROGRAM)
LEARNING OUTCOMES (COMPETENCES)
JOB SEARCH, TRANSITION PHASE, JOB ENTRY
INDIVIDUAL STATE AT TIME OF SURVEY
INDIVIDUAL STATE AT TIME OF SURVEY
JOB SEARCH, TRANSITION PHASE, JOB ENTRY
PROFESSIONAL SUCCESS /
PROFESSIONAL SUCCESS
One time study:
First repetition of study:
1.
thre
sho
ld
2.
thre
sho
ld2
. th
resh
old
Institut für Forschungsinformation und Qualitätssicherung
Example of results: stand. Regression coefficients for Example of results: stand. Regression coefficients for incomeincome
Regression coefficients for monthly income of current occupation
Technical studiesR2=.41
adj. R2=.00 n=105
Medical studiesR2=.25*
adj. R2=.11*n=270
Social sciencesR2=.20*
adj. R2=.08*n=316
Grade of matriculation standard ,179 ,223+ -,203*
Gender (male=1) -,320+ -,020 -,100
Finished vocational training before studying (yes=1) ,317+ ,051 ,114
Duration of stay abroad ,182 ,116 ,031
Part time job during semester break ,093 -,141 ,167
Part time job during lecture period -,063 ,104 -,130
Availability of program specific profiles ,053 ,111 ,057
Social and communication skills ,066 ,101 -,146+
Duration of study -,422* -,004 -,125
Job entrance age - ,028 ,266+
Final degree of study -,165 -,012 ,052
Foreign language skills -,025 ,080 ,106
Proficiency in own subject ,268 ,010 ,127
Ability to develop new ideas and problem solutions -,116 -,115 -,110
Ability to work efficiently towards a goal -,236 ,066 -,089
Ability to critically rethink own ideas and ideas of others
,181 -,042 -,058
Use of personal contacts for job search ,063 -,323** ,011
Employers searched for graduates with other study focus
-,043 - -,076
Importance of family -,119 -,004 ,101
Importance of career ,215 -,209* -,270**
Figure 3: Regression coefficients for the monthly income of current occupation; +/ */ ** = significant on the 10%-/ 5%-/ 1%-level (“-“ = variable was not included in the explanatory model)
Institut für Forschungsinformation und Qualitätssicherung
Example of results: stand. Regression coefficients for Example of results: stand. Regression coefficients for incomeincome
Regression coefficients for monthly income of current occupation
Technical studiesR2=.41n=105
Medical studies
R2=.25*n=270
Social sciencesR2=.20*
n=316
Grade of matriculation standard ,179 ,223+ -,203*
Gender (male=1) -,320+ -,020 -,100
Social and communication skills ,066 ,101 -,146+
Duration of study -,422* -,004 -,125
Job entrance age - ,028 ,266+
Final degree of study -,165 -,012 ,052
Use of personal contacts for job search ,063 -,323** ,011
Importance of family -,119 -,004 ,101
Importance of career ,215 -,209* -,270**
Figure 3: Regression coefficients for the monthly income of current occupation; +/ */ ** = significant on the 10%-/ 5%-/ 1%-level (“-“ = variable was not included in the explanatory model)
• Similar Results in Dresden Graduate Survey with the same analysis scheme, but the social background was more important
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Interpretation of results (Freiburg & Interpretation of results (Freiburg & Dresden)Dresden)• Influencing income: Previous job training, GPA (Secondary school) and
gender, in Medicine: Personal contacts and career oriantation are important
• Influencing job satisfaction: Technical sciences: Socioeconomic background, previous job trainingEngineering: Tendendy for women to be less satisfied Humanities: Higher self-assessed expertise and family orientation: lower job satisfaction
• Influencing job adequacy:Engineering: gender - women have less adequate jobs
Prospects for analysis?
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Further informationFurther information
• IfQ Bonn: www.research-information.de
• Freiburg Graduate Surveys: www.qm.uni-freiburg.de/projekte
• Requirements of graduate surveys for measuring HE outcomes: „Leistungsbewertung, Leistungsanreize und die Qualität der Hochschullehre“ (2007) www.universitätsverlagwebler.de/krempkow.html, Onlinepublikation (2005) http://nbn-resolving.de/urn:nbn:de:swb:14-1129208825969-55860
• Determinants of professional success: Zeitschrift für Evaluation 1/2006 (www.zfev.de), S. 7-37, bzw. http://www.kfbh.de/downloads/Was_macht_Hochschulabsolventen_erfolgreich.pdf
• Dresden Graduate Survey: www.kfbh.de/absolventenstudie
Institut für Forschungsinformation und Qualitätssicherung
Example of results: stand. Regression Example of results: stand. Regression coefficientscoefficients
Recommended