Man In India, 95(2015) © Serials Publications(Special Issue: Researches in Education and Social Sciences)

Address for communication: Mei-Teng Ling and Vincent Pang, Faculty of Psychology and Education,Universiti Malaysia Sabah,88400 Kota Kinabalu, Sabah, MALAYSIA, E-mail: lingmeiteng@gmail.com

RASCH ANALYSIS: PSYCHOMETRIC PROPERTIESOF MALAYSIAN SECONDARY SCHOOL STUDENTS’LEADERSHIP SCALE

Mei-Teng Ling and Vincent Pang

The purpose of this study is to produce empirical evidence of psychometric properties of a surveyquestionnaire, the Malaysian Secondary School Students’ Leadership Scale (M3SLS) Version 2.The scale was analysed by using Rasch Model. The questionnaire was administered on 2340students from 26 government schools in four main divisions of Sabah, Malaysia. The resultsshowed that secondary dimension is noticeable and the constructs were suggested to be analysedseparately in future. PTMEA Corr are all in positive values. Person statistics (separation>2;reliability >0.8) and item statistics (separation>3; reliability >0.8) were within the acceptablerange. All items in the M3SLS were in the range of 0.5 to 1.50, except item V5 (2.01), item V3(1.64), item V4 (1.63), item V1 (1.63) and item V7 (1.53). All the items were maintained after thetest.

Keywords: Validity, Reliability, Leadership Competency, Rasch Measurement

Introduction

Instruments such as tests and questionnaires are usually used to measure students’performances and perceptions in schools. There are a number of instruments usedto assess leadership competencies. Examples of scales that have been developedare Penn State Leadership Competency Inventory (Yoon et al., 2010), AdaptiveLeadership Competency Profile (Sherron, 2000), Prospector Instrument (Spreitzeret al., 1997), A 360-Degree Peer Evaluation of Leadership Competencies Scales(Rogers, 2001), and Multifactor Leadership Questionnaire (MLQ) (Bass and Avolio,1994). However, most of these scales were developed in the western countries.Leadership qualities required in the Malaysian context may be different.Furthermore, extra efforts are needed to translate an instrument into target languageif a researcher uses an instrument which is of different context and the culture. Assuggested by Tuleja et al. (2011), a good questionnaire constructed in single culturemay not necessarily “travel well” across cultures. In Malaysia, most of theresearchers such as Mohd Yusri Ibrahim and Aziz Amin (2014), Bity SalwanaAlias et al. (2010), Yahya Bin Don (2009), Azita (2007) and Fareeza (2010) weremore interested in assessing the leadership competency of principals, assistantprincipals, coaches and teachers rather than students in school. However, secondaryschool students also possess leadership potential (Fertman and Linden, 1999; Hine,2011) and should be concerned so that educators can help students in developing

2 MAN IN INDIA

leadership skills and building character at the earlier stage. The instrument usedin this study is the second version of questionnaire built to assess the level ofleadership competency of secondary school students based on their personality,values and leadership skills. The Rasch Model has been used in a variety ofevaluation settings and provides detailed information on several quality aspectsof a test such as item fit, unidimensionality, and reliability of the items andpersons. Besides, it provides valid measures for tests with missing data (Sköld,Hermansson, Krumlinde-Sundholm, & Eliasson, 2011). The purpose of this studywas to produce empirical evidence of psychometric properties of the MalaysianSecondary School Students’ Leadership Scale (M3SLS) by using RaschMeasurement Model.

Instrument and Method

M3SLS is in a form of questionnaire which contains 68 items of five-point ratingscale that represent three main constructs, namely personality, values and leadershipskills (Tubbs and Schulz, 2006). M3SLS was used to gauge the three constructswhich comprises of personality- P (15 items), values – V (18 items), and Leadershipskills – S (35 items). Item are quantitatively analysed using WINSTEPS to assessthe suitability of items.

Samples

After the instrument was pilot tested, it was distributed to a larger group ofrespondents to ensure the stability of the scale. The instrument was administeredto 2340 students from 26 government secondary schools in four main divisionsof Sabah. 121 questionnaires were dropped due to incomplete andinvalid responses, reducing the number of cases to 2183. They consisted of877 (40.2%) male and 1306 (59.8%) females. Form 2 respondents showedthe highest proportion (34.3%) while Form 1 students showed the lowest(32.7%). The location of the schools were classified as rural or urban area. Theproportion of the respondents from the urban areas is 8% more than that fromrural areas.

Results and Discussions

Unidimensionality

In PCAR analysis, the raw variance explained by measures was 22.3% while theunexplained variance in first contrast was 5.4%. The raw variance explained bymeasures of 22.3% closely matched the expected target of 22.2% (Table 1).Although this meets the minimum requirement of 20% for acceptable calibrationby Reckase (1979), the more stringent threshold of 40%, as suggested by Conradet al., (2011) was not achieved. The reason for not achieving the minimum 40% asrecommended by Linacre (2003) is the disruption of items as noise. The noise

RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES... 3

level was 6.2%, considered as acceptable because it was far from maximum of15% as recommended by Linacre (2003) and Fisher (2007). There were five itemsshowing standard residual correlation values of more than 0.50 (Table 2). Thisindicates that respondents interpreted the items related to the same thing. The itemsshowed infit mean squares of more than 1.5, therefore investigation on the twoitems was done based on the item fitness. Besides, the test was probablyunidimensional when a secondary dimension have the strength of less than 3 items(Linacre, 2003). However, the Eigen value of the unexplained variance in firstcontrast was 5.4, which means that five items fall in secondary dimension in theinstrument. Therefore, each constructs was suggested to analyse separately in thenext study.

TABLE 1: TABLE OF STANDARDIZE RESIDUAL VARIANCE OFM3SLS V.2

Empirical Modeled

Total raw variance in observations = 87.6 100.0% 100.0%

Raw variance explained by measures = 19.6 22.3% 22.2%

Unexplained variance in 1st contrast 5.4 6.2% 8.0%

TABLE 2: NOISE ITEMS IN M3SLS V.2

Contrast Loading Measure Infit MNSQ Outfit MNSQ Item

1 .68 -.38 1.64 1.69 V3

1 .68 -.40 1.63 1.66 V1

1 .61 -.53 1.63 1.69 V4

1 .61 -.29 2.01 2.09 V5

1 .50 -.01 1.53 1.54 V7

Item polarity

The result shows the values of PTMEA Corr in MSLCS version 2 are positive. Thepositive values of PTMEA Corr prove that the items are carefully constructed(Bond and Fox, 2001). The maximum PTMEA Corr is 0.47 for item 19, Item 21and Item 29 in Leadership Skills Construct (Table 3). A high PTMEA Corr indicatesthat an item is able to distinguish between the ability of respondents (Linacre,2003). There is no negative PTMEA Corr found in this study, therefore, it can beconcluded that the items will contribute to the measurement of leadershipcompetency scale. A negative value or zero indicates that the link for the itemresponse or respondents is in conflict with the variable or constructs (Linacre,2003). Furthermore, positive value of PTMEA Corr shows that the items are ableto discriminate or differentiate the level of leadership competencies held by therespondents.

4 MAN IN INDIA

TABLE 3: ANALYSIS OF PTMEA CORR. IN M3SLS VERSION 2

Items PTMEA Items PTMEA Items PTMEA Items PTMEACorr. Corr. Corr. Corr.

CP1 0.36 V3 0.24 LS2 0.42 LS19 0.47CP2 0.41 V4 0.23 LS3 0.37 LS20 0.44CP3 0.39 V5 0.07 LS4 0.33 LS21 0.47CP4 0.21 V6 0.36 LS5 0.43 LS22 0.32CP5 0.39 V7 0.27 LS6 0.46 LS23 0.44CP6 0.30 V8 0.37 LS7 0.36 LS24 0.45CP7 0.37 V9 0.24 LS8 0.40 LS25 0.46CP8 0.43 V10 0.35 LS9 0.43 LS26 0.42CP9 0.23 V11 0.41 LS10 0.43 LS27 0.43CP10 0.40 V12 0.38 LS11 0.41 LS28 0.48CP11 0.37 V13 0.34 LS12 0.42 LS29 0.47CP12 0.43 V14 0.27 LS13 0.41 LS30 0.45CP13 0.46 V15 0.37 LS14 0.44 LS31 0.37CP14 0.45 V16 0.37 LS15 0.41 LS32 0.44CP15 0.44 V17 0.33 LS16 0.38 LS33 0.45V1 0.27 V18 0.30 LS17 0.30 LS34 0.46V2 0.14 LS1 0.37 LS18 0.45 LS35 0.46

Person and Item reliability and separations

The person separation index of M3SLS is 2.90 (Table 4). The person separationindex indicates that the number of strata which measured the leadership competencyof individuals can be divided into three. The person reliability of M3SLS is 0.89.The person reliability is more than 0.8, which is an acceptable and strong value.The item separation index of M3SLS is 15.59 (Table 5), which is good andacceptable, based on Linacre (2003) that the separation index of more than 2 isgood. High item separation (> 3, item reliability > 0.9) implied that the personsample is enough to confirm the item difficulty hierarchy, which is the constructvalidity of the instrument (Linacre and Wright, 2012). The higher the number ofseparations, the more confidence the researcher can place in the replicability ofthe item placement across other samples (Bond and Fox, 2007). Item reliability is1.00, which is acceptable and considered strong because they are more than 0.8.The high item reliability might be due to the wide difficulty range of items and alarge sample size. When the value is high, it indicates that the sample size is enoughfor stable comparisons between items (Linacre and Wright, 2012). Table 5 showsthe summary of item statistics of LCS produced by Winsteps.

The maximum level of item measurement was +0.99 logit (SE: 0.38) whilethe maximum measure of person is +2.45 logit (SE: 0.44). The range of the itemscale that can be exploited were only from -0.80 to +2.45. There was a small gapshowing respondents that were item-free, with a width of 1.46, from +0.99 to+2.45 logit. There were items that were person-free, which shows the item measureat -0.59 logit compared with respondents who were at least competent at -0.80.

RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES... 5

TABLE 4: SUMMARY OF PERSON STATISTICS FOR M3SLS VERSION 2

Raw Count Measure Model Infit Outfitscore error

MNSQ ZSTD MNSQ ZSTD

Mean 178.8 68.0 .66 .13 1.01 -.3 1.02 -.3S.D 25.2 .0 .44 .01 .46 2.8 .47 2.7MAX. 252.0 68.0 2.45 .22 3.18 9.0 3.62 9.0MIN. 99.0 68.0 -.59 .12 .12 -9.6 .12 -9.4

REAL RMSE .14 ADJ.SD .42 SEPARATION 2.90 PERSON RELIABILITY .89MODEL RMSE .13 ADJ.SD .42 SEPARATION 3.18 PERSON RELIABILITY .91S.E. OF PERSON MEAN = .01PERSON RAW SCORE-TO-MEASURE CORRELATION = .99CRONBACH ALPHA (KR-20) PERSON RAW SCORE RELIABILITY = .91

TABLE 5: SUMMARY OF ITEM STATISTICS FOR M3SLS VERSION 2

Raw Count Measure Model Infit Outfitscore error

MNSQ ZSTD MNSQ ZSTD

Mean 5741.5 2183.0 .00 .02 1.01 -1.1 1.02 -1.1S.D 722.6 .0 .38 .00 .28 6.8 .29 6.8MAX. 7104.0 2183.0 .99 .03 2.01 9.9 2.09 9.9MIN. 3729.0 2183.0 -.80 .02 .58 -9.9 .58 -9.9

REAL RMSE .02 ADJ.SD .38 SEPARATION 15.59 PERSON RELIABILITY 1.00MODEL RMSE .02 ADJ.SD .38 SEPARATION 16.49 PERSON RELIABILITY 1.00S.E. OF PERSON MEAN = .05UMEAN=.000 USCALE=1.000ITEM RAW SCORE-TO-MEASURE CORRELATION = -1.0032745 DATA POINTS. LOG-LIKELIHOOD CHI-SQUARE: 379694.36 with 146191 d.f. p=.0000

This situation shows that several items, within 0.29 logit are the standard personality,values and skills that were practiced by the respondents in performing leadershiptasks in school. This instrument is found to have low standard error of measurement(SE), ± 0.02 and quite high person Infit MNSQ SD at 0:46 logit. Both the InfitMNSQ and ZSTD values were close to the ideal values of 1 and 0.

Fit statistics

All items in the M3SLS were in the range 0.5-1.50, except item V5 (2.01), item V3(1.64), item V4 (1.63), item V1 (1.63) and item V7 (1.53). Except for item 20 (V5-Friendship), the item fit map generated by Quest showed all items were fit. Across-plot between person with 63 items and 68 items was carried out to investigatethe effects of the deleted items on the result of the study (Linacre, 2010). None of theobservations were out of the two curves and most of the observations clustered aroundthe Plot-Empirical line (Figure 1). Less than 5% (5/100 x 2183 = 109) of theobservations dropped outside of the straight line (perfect line). This concludes thatthe person measures obtained with 63 items were substantially similar to those obtainedwith 68 items. Therefore, the items were not deleted.

6 MAN IN INDIA

Conclusion

With the examination of the psychometric properties of the M3SLS instrument itwas found that the validity and reliability of the instrument are acceptable to measureleadership competency of secondary school students. It has also fulfilled otherpsychometric properties required of an acceptable instrument. These imply thatM3SLS is a truly leadership competency instrument that can be used to measurethe level of leadership among secondary school students in Malaysia. Themeasurement of leadership level will enable educators to plan and execute variousleadership programs so that students of various levels of leadership can be developedas effective leaders.

References

Amrin, H. A. (2007). Penilaian Pelajar Terhadap Kompetensi Pemimpin Program PendidikanLuar di Institutsi Pendidikan Guru Malaysia (Students’ evaluation on the instructors’competencies in the teaching and learning process of the Outdoor Education Program atthe Malaysian Teacher Education Institutes.). (Unpublished Doctoral Dissertation).Universiti Putra Malaysia, Serdang, Malaysia.

Bass, B. M., & Avolio, B. J. (1994). Improving Organizational Effectiveness ThroughTransformational Leadership. Thousand Oaks, CA: Sage Publications Inc.

Bity Salwana Alias, Ahmad Basri Md. Yussof, Ramlee Mustapha, & Mohammed Sani Ibrahim.(2010). Analisis Kompetensi Pengetua Berdasarkan Kualiti Peribadi , Pengetahuan,Kemahiran dan Amalan dalam Bidang Pengurusan Sekolah Menengah Malaysia. JurnalPendidikan Malaysia, 35(2), 31–41.

Bond, T. G., & Fox, C. M. (2007). Applying the Rasch Model: Fundamental Measurement in theHUman Sciences (2nd ed.). Mahwah, New Jersey: Routledge.

Conrad, K. J., Conrad, K. M., Dennis, M. L., Riley, B. B., & Funk, R. (2011). Validation of theCrime and Violence Scale (CVS) to the Rasch Measurement Model GAIN Methods Report1.2. Evaluation review. Chicago, IL.

Figure 1: Invariance of Person Measures of 63 and 68 items in M3SLS

RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES... 7

Fareeza, A. R. (2010). Assistant Pronsipals Leadership Capacity Building Through The MalaysiaEducation Quality Standards Instrument. (Unpublished Doctoral Dissertation). UniversitiMalaya, Kuala Lumpur, Malaysia.

Fertman, C. I., & Linden, J. A. van. (1999). Character education for developing youth leadership.National Association of Secondary School Principals Bulletin, 83(605), 11–16.

Fisher, J. W. P. (2007). Rating Scale Instrument Quality Criteria. Rasch MeasurementTransactions, 21, 1095.

Hine, G. (2011). Exploring the Development of Student Leadership Potential Within a CatholicSchool: A Qualitative Case Study. (Doctoral Dissertation, University of Notre Dame,Australia). Retrieved from http://researchonline.nd.edu.au/cgi/viewcontent.cgi?article=1062&context=theses

Jackson, T. R., Draugalis, J. R., Slack, M. K., Zachry, W. M., & Agostino, J. D. (2002). Validationof Authentic Performance Assessment/ : A Process Suited for Rasch Modeling. AmericanJournal of Pharmaceutical Education, 66, 233–243.

Linacre, J. M. (2003). Dimensionality: contrasts & variances. Retrieved January 20, 2015, fromhttp://www.winsteps.com/winman/principalcomponents.htm

Linacre, J. M. (2010). When to stop removing items and persons in Rasch misfit analysis? RaschMeasurement Transactions, 23(4), 1241.

Linacre, J. M., & Wright, B. D. (2012). A user’s guide to WINSTEPS ministeps Rasch modelcomputer programs. Chicago: Mesa Press.

Liu, X. F. (2010). Using and Developing Measurement Instruments in Science Education: ARasch Modeling Approach (HC). Charlotte, North Carolina: Information Age Publishing,Inc.

Mohd Yusri Ibrahim, & Aziz Amin. (2014). Model Kepemimpinan Pengajaran Pengetua DanKompetensi Pengajaran Guru. Jurnal Kurikulum & Pengajaran Asia Pasifik, 2(1), 11–25.

Nardi, P. M. (2003). Doing Survey Researc/ : A guide on quantitative method. United States ofAmerica: Pearson Education, Inc.

Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results andimplications. Journal of Educational and Behavioral Statistics, 4(3), 207–230.

Rogers, A. (2001). A 360-Degree Peer Evaluation of Leadership Competencies.(Doctorate Dissertation). Available from ProQuest Dissertations & Theses Global. (UMINo. 3022343).

Rust, J., & Golombok, S. (2009). Modern Psychometrics: The Science of PsychologicalAssessment (3rd ed.). United States of America: Routledge.

Sherron, C. T. (2000). Psychometric Development of The Adaptive Leadership CompetencyProfile. (Doctorate Dissertation). Available from ProQuest Dissertations & Theses Global.(UMI No. 3019211).

Siti Rahayah Ariffin, Bishanani Omar, Anita Isa, & Sharida, S. (2010). Validity and reliabilitymultiple intelligent item using rasch measurement model. Procedia - Social and BehavioralSciences, 9, 729–733. doi:10.1016/j.sbspro.2010.12.225

Sköld, A., Hermansson, L. N., Krumlinde-Sundholm, L., & Eliasson, A.-C. (2011). Developmentand evidence of validity for the Children’s Hand-use Experience Questionnaire (CHEQ).Developmental Medicine and Child Neurology, 53(5), 436–42. doi:10.1111/j.1469-8749.2010.03896.x

8 MAN IN INDIA

Spreitzer, gretchen M., Kizilos, M. A., & Nason, S. W. (1997). A dimensional analysis of therelationship between psychological, empowerment and effectiveness, satisfaction and strain.Journal of Management, 23(5), 679–704.

Tubbs, S., & Schulz, E. (2006). Exploring a taxonomy of global leadership competencies andmeta-competencies. Journal of American Academy of Business, 8(2), 29–35.

Tuleja, E. a., Beamer, L., Shum, C., & Chan, E. K. Y. (2011). Designing and DevelopingQuestionnaires for Translation Tutorial. IEEE Transactions on Professional Communication,54(4), 392–405. doi:10.1109/TPC.2011.2172834

Yahya Bin Don. (2009). Kolerasi dan Pengaruh Kompetensi Emosi terhadap KepimpinanSekolah: Perbandingan antara Sekolah Berkesan dengan Sekolah Kurang Berkesan(Correlation and Effect of Emotional Competence of School Leadership: Comparisonbetween the Effective School and Less Effective School). (Unpublished Doctoral Dissertation).Universiti Malaya, Kuala Lumpur, Malaysia.

Yoon, H. J., Song, J. I. H., Donahue, W. E., & Woodley, K. K. (2010). Leadership competencyinventory/ : a systematic process of developing and validating a leadership competencyscale. Journal of Leadership Studies, 4(3), 39–50. doi:10.1002/jls