Upload
edertds
View
224
Download
0
Embed Size (px)
Citation preview
8/11/2019 principles of survey - part 1.pdf
1/3
S o f t w a r e E n g i n e e r i n g N o t e s v o l 2 6 n o 6 N o v e m b e r 2 0 0 1 P a g e 1 6
A C M S I G S O F T
P r in c i p le s o f S u r v e y R e s e a r c h
P a r t 1 T u r n in g L e m o n s in to L e m o n a d e
Sh ad L aw ren ce Pf leeg er Barb ara A . K i tch en h am
Systems/So f tware , I n c . Dep ar tmen t o f Co m p u ter Sc ien ce
Wa shington, DC, US A Keele Univers i ty , Staf fordshire , UK
s. p f [email protected] barbara@c s, keele .ac , uk
S e t t in g t h e s c e n e
S urveys a re p robab ly the m o s t com m o nly-used re sea rch m e thod
wor ld-wide . S urvey work i s v is ib le no t on ly because we see m any
exam p les o f i t in sof tware eng inee r ing re sea rch , bu t a l so because
we a re o f ten a sked to pa r t i c ipa te in surveys in our p r iva te
capaci ty, as e lectors , consum ers , or service users . Thi s wide-
spread use o f su rveys m ay g ive us the im pres s ion tha t su rvey-
based re sea rch i s s t ra igh t fo rward , an easy op t ion fo r re sea rche rs
to ga the r im por tan t in form a t ion about p roduc t s , con tex t ,
p roces ses, worke rs and m ore . In our pe rsona l expe r ience wi th
apply ing and eva lua t ing re sea rch m e thods and the i r re su l t s , we
ce r ta in ly d id no t expec t to encounte r m a jor p rob lem s wi th a
survey tha t we p lanned , to inves t iga te i s sues a s soc ia ted wi th
technolo gy adopt ion . Th i s a r t i c le and subsequent ones in th i s
se r ie s desc r ibe how w rong we were . We do no t wan t to g ive the
im pres s ion tha t the re i s any way of tu rn ing a bad survey in to a
good one ; i f a su rvey i s a l em on, i t s t ays a l em on. However , we
be l ieve tha t l ea rn ing f rom our m is takes i s the way to m ake
lem on ade f rom lem ons . S o th i s s e r i e s o f a r t i c le s sha res wi th you
our l e s sons l ea rned , in th e hope of im prov ing survey re sea rch in
sof tware eng inee r ing .
W e b e g a n o u r i n v e s t i g a t i o n b y r e v i e w i n g w h a t i s k n o w n a b o u t
technolo gy t rans fe r. We fou nd tha t , a few yea rs ago , Ze lkowi tz ,
Wal lace and Bink ley (1998) surveyed prac t i t ione rs to de te rm ine
the i r co nf idence in d i f fe ren t types o f em pi r ica l eva lua t ions a s the
bas i s fo r t echnolo gy adopt ion dec i s ions . The i r f ind ings conf i rm ed
wh a t we have long suspec ted : the ev idence p roduced by the
resea rch com m u ni ty to suppor t t echnology adopt ion is no t the
k ind o f ev idence be ing sought by p rac t it ione rs. To bu i ld on
Ze lkow i tz e t a l . ' s work , we con tac ted the pub l i she r o f Appl ied
S of tware Deve lopm ent , a news le t t e r whose reade rsh ip inc luded
pr im ar i ly sof tware p ro jec t m anagers . We wanted to do a fo l low-
u p s u r v e y o f m a n a g e r s , t o f i n d o u t w h a t k i n d s o f e v al u a ti o n s t h e y
m a k e o f p r o p o se d t e c h n o l o g ie s , a n d w h a t k i n d s o f e v i d e n c e th e y
re ly on fo r the i r t echnology dec i sions . However , wh en we
a d m i n i s t e r e d o u r s u r ve y
and ana lyzed the re su l t s , we rea l i zed tha t we had m ade e r ro rs in
survey des ign , cons t ruc t ion , adm in i s t ra t ion and ana lys i s tha t
rende red our re su l t s inconc lus ive a t bes t.
Never the le s s . our l em on o f a su rvey gave us som e use fu l ins igh t s
in to the r igh t and wrong ways to do survey re sea rch . Th i s a r t i c le
i s the f i r s t o f s ix pa r t s about how to do use fu l su rvey re sea rch .
We use our ow n work , p lus the work o f seve ral o the r re sea rche rs ,
to i l lus t ra te the a t t rac t ions an d p i t fa l ls o f the survey t echn ique .
T o u n d e r s t a n d h o w s u r v ey s f it i n t h e l a r g e r s c h e m e o f e m p i r i c
inves t iga tion , we inv i t e you to read the s e r i e s we wro te , s t a r t ing
Decem ber 1995 , in S of tware Eng inee r in g Notes . The re ,
desc r ibed s eve ra l em pi r i ca l m e thods , inc lud ing case s tud i
fac to r ana lyses , expe r im ents and surveys , to he lp you dec i
wh ich t echn ique is appropr ia te in wh ich s i tua t ion . Here ,
a s s u m e y o u h a v e d e c i d e d to d o a s u r v e y , a n d w e f o c u s o n h o w
organ ize , adm in i s te r and ana lyze one so tha t you ge t use f
meaningful resul ts .
W h at our se r ies w i ll d iscuss
This f i r s t ins ta l lm ent o f our s e r i e s desc r ibes th ree surveys : t
repor ted in the l i t e ra tu re and the one we a t t em pted recen t ly . W
provide you wi th a gene ra l ove rv iew of each survey ; we wi l l u
pa r t i cu la rs o f each survey in subsequent ins ta l lm e nts a s exam pl
o f th e s t at e o f t h e p r a c ti c e a n d h o w i t m i g h t b e i m p r o v e d . W
also sugges t gene ra l re fe rence m a te r ia l fo r s tudy ing surv
techniques .
S ubsequent ins ta l lm ents wi l l focus on these i s sues :
S urvey des ign , inc lud in g d i s cus s ions o f s am ple s
and re sponse ra te
Ques t ionna i re des ign and cons t ruc t ion
I s sues in survey adm in i s t ra t ion , inc lud i
m ot iva t ing re spondents , su rveyo r b ia s , p re - and p i
t e s t s , su rvey docum enta t ion , and survey re l i ab i l
a n d v a l i d it y
Da ta analys is .
P o p u l a t io n a n d s a m p l i n g
W h a t i s a s u r v e y ?
To beg in , l e t us rev iew exac t ly wh a t a su rvey i s. A survey is n
jus t the ins t rum ent ( the ques t ionna i re o r check l i s t ) fo r ga the r i
in form a t ion . I t i s a com p rehens ive sys tem for co l l ec t i
in form a t ion to desc r ibe , com pare o r exp la in knowledge , a t t i tud
and behav ior . Thus , the survey ins t rum ent i s pa r t o f a l a rg
survey proces s w i th c lea r ly -de f ined ac t iv i t i e s:
1. Set t ing specif ic , mea surab le object ives
2 . P lann in g and s chedul ing the survey
3 . Ensu r ing tha t appropr ia te re sources a re ava i l ab le
4 . Des ign ing the survey
5 . P repa r ing the da ta co l l ec t ion ins t rum ent
8/11/2019 principles of survey - part 1.pdf
2/3
S o f t w a r e E n g i n e e r i n g N o t e s v o l 2 6 n o 6 N o v e m b e r 2 0 0 1 P a g e 1 7
CMSIGSOFT
6 . V a l i d a t i n g t h e i n s t r u m e n t
7 . S e l ec t i ng pa r t i c i pam s
8 . A d m i n i s t e ri n g a n d sc o r i n g t h e i n s t r u m e n t
9 . A n a l y z i n g t h e d a t a
10 . Rep or t i ng t he r e su l t s .
T he r e a r e s eve r a l d i f f e r en t t ypes o f su r veys . S ur veys can be e i t he r
supe r v i sed o r no t , depend i ng on t he ob j ec t i ves and t he r e sour ces
ava i l ab l e . I f supe r v i sed , we can a s s i gn one su r vey r e sea r che r t o
e a c h r e s p o n d e n t , t o e n s u r e t h a t t h e r e s p o n d e n t u n d e r s t a n d s e a c h
q u e s t i o n a n d p r o v i d e s a n a n s w e r . T e l e p h o n e i m e r v i e w s a re o f te n
o f t h is t y p e , w h e r e a q u e s t i o n e r w o r k s o n e - o n - o n e w i t h a
r e s p o n d e m t o e l i c it a n s w e rs . O r a s u r v e y c a n b e a d m i n i s te r e d t o a
gr oup , w i t h a su r vey r e sea r che r ava i l ab l e t o c l a r i f y and e l abor a t e
o n t h e i n s t r u c ti o n s i n t h e s u r v e y i n s t ru m e m . S o m e s u r v e y s a r e
semi - supe r v i sed , wher e a r e se a r che r exp l a i ns t he ob j ec t i ves and
f o r m a t , p e r h a p s w o r k i n g t h r o u g h s o m e s a m p l e q u e s t io n s , b u t t h e n
l e a v e s th e r e s p o n d e m s t o p r o v i d e i n f o r m a t i o n o n t h e i r o w n .
W e a r e a l l f a m i l i a r w i t h u n s u p e r v i s e d s u r ve y s : t h e a u t o m a t e d -
v o i c e t e l e p h o n e c a l l s i n t h e m i d d l e o f d in n e r , o r t h e m a i l e d
q u e s t i o n n a i r e ( s o m e t i m e s w i t h a p r i z e o r a p o u n d c o i n ) r e q u e st i n g
r esponses abou t l i f e s t y l e , t r ave l hab i t s , o r f ood p r e f e r ences , f o r
e x a m p l e .
A s n o t e d a b o v e , t h e f i r s t s te p i n b e g i n n i n g a n y s u r v e y r e s e a rc h ( o r
any r e sea r ch , f o r t ha t ma t t ed) i s s e t t i ng ob j ec t ives . E ac h
ob j ec t i ve is s i mpl y a s t a t em ent o f t he su r vey ' s exp ec t ed ou t comes .
F or i ns t ance , a su r vey may hope t o i den t i f y t he mos t use f u l
f e a t u re s o f a f r o m - e n d d e v e l o p m e n t t oo l , o r th e m o s t c o m m o n
t r a i n i n g n e e d s f o r n e w h i r e s . I t i s v e r y i m p o r t a n t t h a t t h e
s t a t em em of ob j ec t i ves i nc l ude de f i n i t i ons o f a ll po t en t i a l l y
a m b i g u o u s t e rm s . T h e s e d e f in i t io n s d r i v e n o t o n l y t h e
i n s t r u m e n t ' s d e s i g n a n d d e v e l o p m e n t b u t a l so t h e r e s p o n d e n t s '
u n d e r s t a n d i n g o f th e t e r m i n o l o g y .
W h e r e d o t h e se o b j e ct iv e s c o m e f r o m ? S o m e t i m e s t h e y a r e
d e r i v e d f r o m a p e r c e i v e d n ee d . S o m e o n e i n y o u r o r g a n i z a t i o n
w a n t s t o k n o w s o m e t h i n g a b o u t t h e o r g a n i z a t io n , t h e s t a t e o f t h e
pr ac t i ce , t he e f f ec t i venes s o f a new t echno l ogy , o r the e xpe r i ence
base o f i t s mem ber s , f o r examp l e . A t o t he r t i mes , an ob j ec ti ve
m a y b e d e t e r m i n e d b y a l i t e r a t u r e s e a r c h t o f i n d o u t w h o i s
w r i t i n g a b o u t w h i c h n e w t e c h n o l o g i e s a n d w h e r e t h e r e a r e g a p s i n
cur r en t know l edge . E xp er t s can a l so p r ov i de ob j ec t ives . F or
e x a m p l e , r e s e a r c h e r s p l a n n i n g a s u r v e y a b o u t s o f tw a r e t e st i n g
t echno l og i es may i n i t i a l l y canvass t e s t i ng expe r t s t o de t e r mi ne
w h a t i n f o r m a t i o n is n e e d e d a b o u t w h i c h te c h n o l o g ie s . T h i s k i n d
of so l i c i ta t i on i s somet i mes don e i n a f ocus g r oup , wher e
pa r t i c i pan t s b r a i ns t o r m abou t f u t u r e poss i b i l i t i e s o r d i f f e r en t
o u t c o m e s . W h e n t h e r e i s d i s a g r e e m e n t a b o u t w h a t t h e o b j e ct iv e s
s h o u l d b e , s o m e t i m e s w e f o r m a c o n s e n s u s p a n e l t o e n c o u r a g e
p e o p l e t o c o n v e r g e o n o n e o r t w o i d ea s .
I t i s e s sen t ia l t o devo t e enoug h t i m e t o t he ob j ec t i ves so t ha t t hey
a r e c l ea r an d m easur ab l e , becau se t he ob j ec t ives a r e e s sen t i a l f o r
a l l subsequen t su r vey p r oces s ac t i v i ti e s . F i r s t , t he ob j ec ti ves
d e t e r m i n e w h a t t h e s u r v e y w i ll a s k , o f w h a t p o p u l a t io n , a n d w h a t
i n f o r m a t i on w i l l be co l lec t ed . S econd , t he ob j ec t i ves a r e usua l
r e p h r a s e d a s r e s e a r c h q u e s t i o n s o r h y p o t h e s e s , a n d t h e y h e l p
s u g g e s t w h i c h a r e t h e d e p e n d e n t a n d i n d e p e n d e n t v a r i a b le s o f t
i nves t i ga ti on . T h i r d , t he ob j ec t ives con f i r m whe t h e r a su r vey
t he appr opr i a t e t ype o f emp i r i ca l i nves t i ga t ion . T h a t i s , i f t
ob j ec t i ves a r e f uzzy , t hen we may r equ i r e a d i f f e r en t r e sea r
me t hod .
Descr iption o f our su rvey
T o see how ea sy i t can be t o f a l l in t o s eve r a l su r ve y tr aps , i t
u s e f u l t o u n d e r s t a n d t h e e v o l u ti o n o f o u r o w n s u r v e y . W e h
n o t i ce d t h a t m a n y n e w s l e tt e rs o f t e n i n c l u d e r e a d e r s u r v e y f o r m
s o m e o f w h o s e q u e s t i o n s a n d a n s w e r s c o u l d p r o v i d e u s e f u l i n s i g
i n t o m a n a g e r s ' d e c i s i o n - m a k i n g p r o c e ss e s . W e a p p r o a c h e d t
p u b l i s h e r o f
Applied Software Development;
h e w a s e a g e r
c o o p e r a t e w i t h t h e r e s e a r c h c o m m u n i t y , a n d h e a g r e e d t o i n s e r t
o n e - p a g e s u r v e y i n t h e n e w s l e t t e r a n d g a t h e r t h e r e s p o n s e s f o r u
As a r e su l t , we t ook t he f o l l owi ng s t eps :
1 . W e d e s i g n e d a s u r v e y f o r m a n d a s k e d s e v e r a l o f o
co l l eagues t o c r i t i que i t. T h e su r vey a sk
r e s p o n d e n t s t o e x a m i n e a l is t o f t e c h n o l o g i e s a n d t
u s i f t h e t e c h n o l o g y h a d b e e n e v a l u a t e d a n d i f i t h
b e e n u s e d. I f i t h a d b e e n e v a l u a t e d , w e a s k e d t
r e s p o n d e n t s t o d i s t in g u i s h b e t w e e n a s o
eva l ua t i on , such a s a su r vey o r f ea t u r e ana l ys i s , a
a h a r d e v a l u a ti o n , s u c h a s f o r m a l e x p e r i m e n t
case s t udy .
2 . W e t e s t e d t h e r e s u l ti n g s u r v e y f o r m o n a c o l l e a g
a t L u c e n t T e c h n o l o g i e s . W e a s k e d h i m t o f i ll o
t h e s u r v e y fo r m a n d g i v e u s f e e d b a c k o n t h e c l a r
o f th e q u e s t i o n s a n d r e s p o n se s , a n d o n t h e t i m e
t o o k h i m t o c o m p l e t e th e f o r m . B a s e d o n h i s v e
pos i t i ve r eac t i on t o our ques t i onna i r e , we submi t t
a s l i gh t l y r ev i sed su r vey to t he n ews l e t t e r pub l i she r
3 . T h e pub l i she r t hen r ev i sed our su r vey , sub j ec t t o o
appr ova l , so t ha t i t woul d f i t on on e page o f h
newslet ter .
4 . T h e s u r v e y f o r m w a s i n c l u d e d i n a l l c o p ie s o f
s u m m e r 1 9 9 9 i s s u e o f Applied Softwa
Development.
O f t h e s e v e r al t h o u s a n d p o s s i b le r e c i p i e n t s o f
App lied Softwa
Development
o n l y 1 71 r e s p o n d e d b y s e n d i n g t h e i r s u r v e y f o
back ; t hus , our re sponse r a t e was l ow , whi ch i s typ i ca l i n t h
t y p e o f s u r v ey . T h e s t a f f a t
App lied Software Developm
t r ans f e r r ed t he da t a f r om t he su r vey shee t s t o a sp r eadshee
w h i c h b e c a m e t h e b a s is f o r o u r a n a l y s i s.
Descr ipt ion of the Lethbridge surv ey
I n 1 9 9 8 , L e t h b r i d g e ( 1 9 9 8 a n d 2 0 0 0 ) c o n d u c t e d s u r v e y s t o h e
h i m u n d e r s t a n d t h o s e a r e a s w h e r e p r a c t i t i o n e r s f e e l t h e y n e
m o r e o r b e t t e r e d u c a t io n . T h e g o a l o f t h e s u r v e y s w a s t o p r o v
i n f o r m a t i o n t o e d u c a t i o n a l i n s t i t u t i o n s a n d c o m p a n i e s a s t h
8/11/2019 principles of survey - part 1.pdf
3/3
A C M S I G S O F T S o f t w a r e E n g i n e e ri n g N o t e s v o l 2 6 n o 6 N o v e m b e r 2 0 0 1 P a g e 1 8
plan curr icula and t rain ing programs. A secondary goal involved
providing da ta that w i l l assi s t educators and pract it ioners in
evaluat ing e xis t ing and propose d curricula.
Lethbr idge and his team recrui ted par t icipants for the surveys in
two ways: by approaching com panies d i rectly and asking them to
par t icipate, and by adver t i s ing for par t icipants on the Web. To
determ ine the e f fects of formal educat ion, Lethbr idge presented
the responden ts wi th a l i s t of topics related to comp uter science,
mathe mat ics and business . For each topic, the respondent was
asked How m uch d i d you learn abou t t h is i n your fo rmal
educat ion? The choices for answe rs ranged on a s ix-point Liker t
scale f rom learned nothing to learne d in depth . Other
quest ions included
W hat i s your current know ledge about th i s
cons i der ing w hat you have l earned on t he j ob as w el l
as forgot ten?
Ho w useful has th is speci f ic mater ial been to you in
your career?
Ho w useful wou ld i t be (or have been) to learn mo re
about th i s (e.g . addi t ional courses)?
Ho w mu ch inf luence has learning the mater ial had
on yo ur th inking ( i .e . your approach to problem s and
your general maturi ty) , whethe r or not you have
di rect ly used the detai l s of the m ater ial? Please
consider inf luence on both your career and other
aspects of you l ife.
Descript ion o f the Finnish study
Recent ly , Ropponen and Lyyt inen (2000) descr ibed thei r
exanf inat ion of r i sk ma nagem ent practices. The y adm inistered a
survey addressing two ove ral l questions:
* W hat a re t he componen t s o f so f tware deve l opmen t
r i sk?
Wha t r i sk managem en t p rac t ices and env i ronmen t a l
cont ingencies help to address these components?
To f ind out the answe rs , the researchers m ai led a quest ionnai re to
each o f a p re- se l ec t ed sampl e o f members o f t he F i nn i sh
Informat ion Processing Associat ion whose job t i tl e was
'manager or equivalent. How ever , they sent the quest ionnai re to
at most two mana gers in the sam e company.
Ropponen and Lyyt inen asked twenty quest ions about r i sk by
present ing scenar ios and asking the respondents to rate thei r
occurrence wi th a f ive-point Liker t scale, ranging f rom hardly
ever to almost always. For examp le, the scenar ios included
Yo ur project i s canceled before complet ing i t .
and
Subco nt racted tasks in the project are performed as expected.
The researchers posed addi t ional quest ions relat ing to
organizat ional character i s t ics , such as the organizat ion ' s s ize,
industry , type of systems developed, and
contr ctu l
ar r angemen
They also sought technology character i s t ics , such as the newne
of the technology, the com plexi ty and n ovel ty of technologic
solut ions, and the process technologies used. Final ly , they ask
quest ions about the responden ts themselves: thei r exper ien
wi th d i f ferent s izes of projects , thei r educ at ion, the so f tware use
and exper ience wi th project management .
References for survey techniqu es
Subsequent instal lments of th i s ser ies wi l l focus on surv
research in sof tware engineer ing. To f ind mo re gener
informat ion about surveys, we recommend the fo l lowi
publications.
Arlene Fink,
The Survey Handbook
Sage Publ icat ions, 1995.
Th i s comprehens i ve book exp l a i ns how t o ask su rv
questions, how to design survey s, how to sample
surveys, how to analyze survey data, and how to repo
on surveys.
Li nda Bourque and Eve F i e l der ,
How to Conduct Se
administered and Mail Surveys
Sage Publ icat ions, 1995.
This book looks in detai l at sel f -administered and ma
surveys.
Mark Li twin,
How to Measure Survey Reliabil i ty and Validit
Sage Publications, 1995.
This book def ines survey rel iabi l i ty and val id i ty a
exp l a ins how t o m easu re t hem.
Bibliography
Let hb ri dge , T i mot hy , A su rvey o f t he r e l evance o f com pu t
science and sof tware engineer ing educat ion,
Proceedings o f t
llth International Conference on Software Engineering
I E E
Com puter Society Press , 1998.
Lethbr idge, T imothy, W hat knowledg e i s impo rtant to a sof twa
professional ,
IEE E Computer
May 2000.
Ropponen , J . and K. Lyy t inen , Com ponen t s o f so f t wa
devel opmen t r i sk : How t o add ress t hem. A p ro j ec t manag
survey,
IEEE Transactions on Software Engineering
26(2
February 2000.
Zelkowi tz, Marvin V. , Dolores R. Wal lace and David Binkle
'Understanding the cul ture clash in sof twa re engineer in
technology t ransfer , Universi ty of M aryland tec hnical repor t ,
June 1998.