View
219
Download
2
Category
Tags:
Preview:
Citation preview
Software Engineering Research Approaches
Balancing theory and praxisHow engineering research differs
from scientific researchThe role of empirical studiesModels for SE research
The need to link research with practice
Why after 25 years of SE has SE research failed to influence industrial practice and the quality of resulting software?
Potts argues that this failure is caused by treating research and its application by industry as separate, sequential activities.
What he calls the research-then-transfer approach. The solution he proposes is the industry-as-laboratory approach.
.
Colin Potts, Software Engineering Research Revisited, IEEE Software, September 1993
Research-then-Transfer
Incremental Refinementof research solutions
Problem evolves invisibly to the research community
ProblemV1
Wide gulf bridgedby indirect, anecdotal knowledge
ResearchSolutionV1
Technologytransfer Gapbridged byhard, but frequentlyinappropriate technology
ProblemV2
ProblemV3
ProblemV4
ResearchSolutionV2
ResearchSolutionV3Research
SolutionV4
Research-then-Transfer Problems
Both research and practice evolve separately
Match between current problems in industry and research solutions is haphazard
No winners
Disadvantages of Research-then-Transfer Research problems described and understood in terms of
solution technology - whatever is current research fashion. Connection to practice is tenuous.
Concentration is on technical refinement of research solution - OK but lacks industrial need as focus, so effort may be misplaced.
Evaluation is difficult as research solutions may use technology that is not commonly used in industry
Delay in evaluation means problem researchers are solving has often evolved through changes in business practice, technology etc.
Transfer is difficult because industry has little basis for confidence in proposed research solution.
Industry-as-Laboratory Approach to SE research
ProblemV2
Problem
V1
Problem
V3
Problem
V4
ResearchSolution
V1
ResearchSolution
V2
ResearchSolution
V3
ResearchSolution
V4
Advantages of Industry-as-Laboratory Approach
Stronger connection at start because knowledge of problem is acquired from the real practitioners in industry, often industrial partners in a research consortium.
Connection is strengthened by practitioners and researchers constantly interacting to develop the solution
Early evaluation and usage by industry lessens the Technology Transfer Gap.
Reliance on Empirical Research shift from solution-driven SE to problem-focused SE solve problems that really do matter to practitioners
Early SEI industrial survey research
What a SEI survey* learned from industry: There was a thin spread of domain knowledge in most
projects Customer requirements were extremely volatile.
These findings point towards research combining work on requirements engineering with reuse - instead of the approach of researching these topics by separate SE research communities - as is still found today!
*From ‘A field study of the Software Development Process
for Large Systems’, CACM, November 1988.
Further Results from Potts et al Early 90s Survey23 software development organizations (during 1990-92).
(Survey focused on Requirements Modeling process)
Requirements were invented not elicited. Most development is maintenance. Most specification is incremental. Domain knowledge is important. There is a gulf between the developer and user User-interface requirements continually change. There is a preference for office-automation tools
over CASE tools to support development. I.e. developers found using a WP + DB more useful than any CASE tools.
Industry-as-Laboratory emphasizes Real Case Studies
Advantages of case studies over studying problems in research lab.
Scale and complexity - small, simple (even simplistic) cases avoided - these often bear little relation to real problems.
Unpredictability - assumptions thrown out as researchers learn more about real problems
Dynamism - a ‘real’ case study is more vital than a textbook account
The real-world complications of industrial case studies are more likely to throw up representative problems and phenomena than research laboratory examples influenced by the researchers’ preconceptions.
Need to consider Human/Social Context in SE research
Not all solutions in software engineering are solely technical.
There is a need to examine organizational, social and cognitive factors systematically as well.
Many problems are “people problems”, and require “people-orientated” solutions.
Theoretical SE researchWhile there is still a place for innovative, purely
speculative research in Software Engineering, research which studies real problems in partnership with industry needs to be given a higher profile.
These various forms of research ideally complement one another.
Neither is particularly successful if it ignores the other.Too industrially focused research may lack adequate
theory!
Academically focused research may miss the practice!
Research models for SE
Problem highlighted by Glass*: Most SE Research in 1990s was Advocacy
Research. Better research models needed.
The software crisis provided the platform on which most 90s research was founded.
SE Research ignored practice, for the most part; lack of practical application and evaluation were gapping holes in most SE research.
Appropriate research models for SE are needed.* Robert Glass, The Software -Research Crisis, IEEE Software, November 1994
Methods underlying Models
Scientific method
Engineering method
Empirical method
Analytical methodFrom W.R.Adrion, Research Methodology in Software
Engineering, ACM SE Notes, Jan. 1993
Scientific method
Observe real world
Propose a model or theoryof some real world phenomena
Measure and analyzeabove
Validate hypotheses of the model or theory
If possible, repeat
Engineering method
Observe existing solutions
Propose better solutions
Build or develop bettersolution
Measure, analyze, andevaluate
Repeat until no further improvements are possible
Empirical method
Propose a model
Develop statistical or otherbasis for the model
Apply to case studies
Measure and analyze
Validate and then repeat
Analytical method
Propose a formal theory or set ofaxioms
Develop a theory
Derive results
If possible, compare withempirical observations
Refine theory if necessary
Need to move away from purely analytical method
The analytical method was the most widely used in mid-90s SE research, but the others need to be considered and may be more appropriate in some SE research.
Good research practice combines elements on all these approaches.
4 important phases for any
SE research project (Glass)
Informational phase - Gather or aggregate information via reflection literature survey people/organization survey case studies
Propositional phase - Propose and build hypothesis, method or algorithm, model, theory or solution
Analytical phase - Analyze and explore proposal leading to demonstration and/or formulation of principle or theory
Evaluation phase - Evaluate proposal or analytic findings by means of experimentation (controlled) or observation (uncontrolled, such as case study or protocol analysis) leading to a substantiated model, principle, or theory.
Software Engineering Research Approaches
The Industry-as-Laboratory approach links theory and praxis
Engineering research aims to improve existing processes and/or products
Empirical studies are needed to validate Software Engineering research
Models for SE research need to shift from the analytic to empirical.
Recommended