Embed Size (px)
Citation preview
Overview
• Yet another schedule change
• Systematic Literature Reviews
• “Qualitative” Thesis Topics
• Zotero
Yet another schedule change
Schedule change for Literature Review chapter
• Previously, I gave 1/March/2013 as the deadline for submitting the literature review chapter.
• I am scrapping this deadline.
• Please make an agreement with your thesis supervisor on a schedule for submitting your literature review chapter.
Systematic Literature Reviews (SLRs)
Source of how to do systematic literature reviews (SLRs)• Guidelines for performing SLRs in Software Engineering, Version
2.3, EBSE Technical Report, EBSE-2007-01
• http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.471
• (a PDF copy is in the Thesis Writing Course Web pages)
Definition of an SLR
• A form of secondary study that uses a well-defined methodology to identify, analyse and interpret all available evidence related to a specific research question in a way that is unbiased and (to a degree) repeatable.
• A means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest.
• Individual studies contributing to a systematic review are called primary studies; a systematic review is a form of secondary study.
Glossary of terms associated with SLRs
• Primary study. (In the context of evidence) • An empirical study investigating a specific research question.
• Secondary study. • A study that reviews all the primary studies relating to a specific research
question with the aim of integrating/synthesising evidence related to a specific research question.
• Meta-analysis. • A form of secondary study where research synthesis is based on quantitative
statistical methods.
Glossary of terms associated with SLRs
• Sensitivity analysis. • An analysis procedure aimed at assessing whether the results of an SLR are
unduly influenced by a small number of studies.
• It involves 1. assessing the impact of high leverage studies (e.g. large studies or studies with atypical
results), and
2. ensuring that overall results of an SLR remain the same if low quality studies or high quality studies are omitted from the analysis, or analysed separately.
Glossary of terms associated with SLRs
• Systematic review protocol. • A plan that describes the conduct of a proposed systematic literature review.
• Systematic mapping study (also referred to as a scoping study). • A broad review of primary studies in a specific topic area that aims to identify
what evidence is available on the topic.
• Tertiary study (also called a tertiary review). • A review of secondary studies related to the same research question.
Reasons for Performing Systematic Literature Reviews • To summarise the existing evidence concerning a treatment or technology
• E.g. to summarise the empirical evidence of the benefits and limitations of a specific agile method.
• To identify any gaps in current research in order to suggest areas for further investigation.
• To provide a framework/background in order to appropriately position new research activities.
• However, SLRs can also be undertaken to examine the extent to which empirical evidence supports/contradicts theoretical hypotheses, or even to assist the generation of new hypotheses.
(Section 2.1, page 3)
Motivation for SLRs• Unless a literature review is thorough and fair, it is of little scientific
value.
• SLRs must be undertaken in accordance with a predefined search strategy.
• The search strategy must allow the completeness of the search to be assessed.
• Researchers performing an SLR must make every effort to identify and report research that does not support their preferred research hypothesis as well as identifying and reporting research that supports it.
(Section 2.2, page 3)
Advantages of SLRs
• The well-defined methodology makes it less likely that the results of the literature are biased.• It does not, however, protect against publication bias in the primary studies.
• SLRs can provide information about the effects of some phenomenon across a wide range of settings and empirical methods. • If studies give consistent results, systematic reviews provide evidence that the
phenomenon is robust and transferable.
• If the studies give inconsistent results, sources of variation can be studied.
(Section 2.3, page 4)
Advantages of SLRs
• In the case of quantitative studies, it is possible to combine data using meta-analytic techniques. • This increases the likelihood of detecting real effects that individual smaller
studies are unable to detect.
(Section 2.3, page 4)
Disadvantages of SLRs
• They require considerably more effort than traditional literature reviews.
• In addition, increased power for meta-analysis can also be a disadvantage, since it is possible to detect small biases as well as true effects.
(Section 2.3, page 4)
Distinguishing Features of SLRs• SLRs start by defining a review protocol that specifies
a) the research question being addressed and b) the methods that will be used to perform the review.
• SLRs are based on a defined search strategy that aims to detect as much of the relevant literature as possible.
• SLRs document their search strategy so that readers can assess their rigour and the completeness and repeatability of the process.
• SLRs require explicit inclusion and exclusion criteria to assess each potential primary study.
• SLRs specify the information to be obtained from each primary study including quality criteria by which to evaluate each primary study.
(Section 2.4, page 4)
Example of an SLR procedure
• The diagram on the next slide comes from:
B. Cornelissen, A. Zaidman, A. Van Deursen, L. Moonen, R. Koschke,
A Systematic Survey of Program Comprehension through Dynamic Analysis, IEEE Transactions on Software Engineering: 35(5): 684-702, 2009.
Other types of review: Systematic Mapping Studies• If, during the initial examination of a domain prior to commissioning a
systematic review, it is discovered that very little evidence is likely to exist or that the topic is very broad then a systematic mapping study may be a more appropriate exercise than a systematic review.
• A systematic mapping study allows the evidence in a domain to be plotted at a high level of granularity.
• This allows for the identification of evidence clusters and evidence deserts to direct the focus of future systematic reviews and to identify areas for more primary studies to be conducted.
(Section 2.5.1, page 5)
Some Examples of SLRs
• See the course Web page, Session 10, for examples.
• SLR_GreenMetrisc.pdf
• SLR_unpackingGreenInfoSystems.pdf
Quantitative versus Qualitative Thesis Topics
Qualitative Research In Computer Science
• The next few slides are based closely on the following paper:
Myers, M. D. "Qualitative Research in Information Systems," MIS Quarterly (21:2), June 1997, pp. 241-242. MISQ Discovery, archival version, June 1997, http://www.misq.org/supplements/. Association for Information Systems (AISWorld) Section on Qualitative Research in Information Systems, updated version, last modified: November 5, 2012 www.qual.auckland.ac.nz
Introduction
• Qualitative research involves the use of qualitative data, such as interviews, documents, and participant observation data, to understand and explain social phenomena.
• Qualitative researchers can be found in many disciplines and fields, using a variety of approaches, methods and techniques.
• In Information Systems researchers study the managerial and organizational issues associated with innovations in information and communications technology.
Quantitative Research and Qualitative Research
Quantitative research methods
• Developed in the natural sciences to study natural phenomena.
• Examples: survey methods, laboratory experiments, formal methods (e.g. econometrics) and numerical methods such as mathematical modeling
Qualitative research methods
• Developed in the social sciences to enable researchers to study social and cultural phenomena.
• Examples: action research, case study research and ethnography.
• Qualitative data sources include observation and participant observation (fieldwork), interviews and questionnaires, documents and texts, and the researcher's impressions and reactions
“Triangulation”
• Although most researchers do either quantitative or qualitative research work, some researchers have suggested combining one or more research methods in the one study (called triangulation).
Motivation: why qualitative rather than quantitative?• Qualitative research methods are designed to help researchers
understand people and the social and cultural contexts within which they work.
• The goal of understanding a phenomenon from the point of view of the participants and its particular social and institutional context is largely lost when textual data are quantified
Case Study Research • Case study research is the most common qualitative method used in
information systems.
• A case study is an empirical inquiry that: • investigates a contemporary phenomenon within its real-life context,
especially when the boundaries between phenomenon and context are not clearly evident.
• The case study research method suits, for example, Information Systems research, where the object is the study of information systems in organizations, and interest is focussed on organizational rather than technical issues.
• See the paper for references
5 misunderstandings about case studiesFlyvberg, Qualitative Inquiry, 12, 2 219-245 (2006)
(a) theoretical knowledge is more valuable than practical knowledge;
(b) one cannot generalize from a single case, therefore, the single-case study cannot contribute to scientific development;
(c) the case study is most useful for generating hypotheses, whereas other methods are more suitable for hypotheses testing and theory building;
(d) the case study contains a bias toward verification; and
(e) it is often difficult to summarize specific case studies.
Zotero
