Introduction to Design Science methodologyroelw/Introduction... · Research problemsin design science • “Design a DoA estimation system for satellite TV reception in a car.”

Introduction to Design ScienceMethodology

Prof. Dr. Roel WieringaUniversity of TwenteThe Netherlands

26‐11‐2014 Hanze Hogeschool 1

• R.J. Wieringa. Design Science Methodology forInformation Systems and Software Engineering. Springer, 2014. http://link.springer.com/book/10.1007/978‐3‐662‐43839‐8

• Slides and other material at http://wwwhome.ewi.utwente.nl/~roelw/


Outline

• Design science– Design cycle– Empirical cycle

• Theories– Structure– Function


Outline




• Design science is the design and investigation of artifacts in context

526‐11‐2014 Hanze Hogeschool



7

Subjects of design science

Artifact:

SW component/system,HW component/system,Organization,Business process,Service,Method, Technique,Conceptual structure, ...

Problem context:

SW components & systems, HW components & systems,Organizations,Business processes, Services,Methods, Techniques,Conceptual structures, People,Values, Desires, Fears, Goals, Norms, Budgets,...

Interaction

Something to be designedSomething to be influenced26‐11‐2014 Hanze Hogeschool



Research problems in design science

• “Design a DoA estimation system for satellite TV reception in a car.”

• “Design a multi‐agent aircraft taxi‐route planning system for use on airports”

• “Design an assurance method for data location compliance for CSPs”

• “Is the DoA estimation accurate enough?”

• “Is this agent routing algorithm deadlock‐free?”

• “Is the method usable and useful for cloud service providers?

9

To design an artifactto improve a

problem context

To answer knowledgequestions about the artifact in

context

Problems & Artifactsto investigate

Knowledge,Design problems

The design researcher iterates over these two activities


Design science

10

The context of design research

Improvement design Answering knowledge questions

Social context:Location of stakeholders

Knowledge context:Mathematics, social science, natural science, design science, design specifications, useful facts, practical knowledge, common sense, other beliefs

Existing problem‐solving knowledge, Old designs

Existing answers to knowledge questions

Goals, budgets Designs

New problem‐solving knowledge, New designs

New answers to knowledge questions


Roel1

Slide 10

Roel1 kniowledge base --- knowledge context

also:make a diagram of interaction context, social context, knowledge contextRoel Wieringa; 23-11-2012

Template for design problems

• Improve <problem context> • by <treating it with a (re)designed artifact> • such that <artifact requirements>• in order to <stakeholder goals>

• Improve my health • by taking a medicine • such that my headache disappears • in order for me to get back to work

• Improve the course admin• By integrating with student admin• Such that less errors• In order to reduce workload


Knowledge questions• Descriptive questions:

– What happened?– When?– Where?– What components were involved?– Who was involved?– etc.

• Explanatory questions: – Why? – What has caused the phenomena? – Which mechanisms produced the phenomena? – For what reasons did people do this?


Example knowledge questions• Effect:

– What is the execution time of the DoA algorithm?

– What is its accuracy?

• Trade‐off:– Comparison between algorithms

on these two variables– Comparison between versions of

one algorithm

• Sensitivity:– Assumptions about car speed?– Assumptions about processor?

• Stakeholders:― Who are affected by the DoA

algorithm?

• Goals:– What are their goals?

• Contribution evaluation(after DOA algorithm is in use)– How well does the DoA

algorithm contribute to these goals?


Effect questions

– Central effect question• Effect question: Context X Artifact→ Effects?

– Generalizations• Trade‐off question: Context X Alternative artifact → Effects?• Sensitivity question: Other context X artifact → Effects?

– Trade‐off and sensitivity analysis • Generalize to a larger class of artifacts and contexts• Compare different artefacts• Uncover context assumptions


Contribution questions– Preliminary questions:

• Stakeholder question: Who are the stakeholders?• Goal question: What are their goals?

– Central contribution question:• Contribution question: Do Effects contribute to Stakeholder

goals?


Outline





Implementation evaluation starts a new cycle

Implementation evaluation = Problem investigation

Treatment designDesign validation

Designimplementation

•Stakeholders? Goals? •Conceptual problem framework?•Phenomena? Causes? Effects?•Effects contribute to Goals?

•Specify requirements!•Requirements contribute to goals?•Available treatments?•Design new ones!

•Context & Artifact → Effects?•Effects satisfy Requirements?•Trade‐offs for different artifacts?•Sensitivity for different Contexts?

Legend: Knowledge questions? Actions!


Design research projects

• Design research projects iterate one or more times throughthe design cycle.

Implementation evaluation = Problem investigation

Treatment designDesign validation

Designimplementation

Design cycleskips introduction in original problemcontext

Concurrent engineering• Development may be organized concurrently with successive

versions of the artifact


Time

Problem investigationTreatment designDesign validationImplementation

Evaluation

Tasks

Systems engineering

• Cycles of systems engineering– High level goals, high level requirements– Iterative refinement until– Low‐level approved interfaces, low‐level implementedspecs.

• Shown on next slide



• Iteratively reduce uncertainty about the problem• Once the goals are clear enough, reduce risk of choosing the wrong treatment

Ill‐understood problem

Better understood problem

Treatment idea

Validation

Even betterunderstood problem

Treatment specification

Validation

Still betterunderstood problem

OperationalTreatmentspecification

Validation

Goals and requirementsOperational concept

Feasibility

Prototype

Time

Clear problem, clear goals Solution1 spec Validation Implementation1 Eval

Clear goals, risky treatment Solution2 spec Validation Implementation2 Eval

Clear goals, acceptable risk Solution3 spec Validation Implementation3 Eval

Early requirements Validation

Process versus logic• Process is the performance of tasks• Logic is the structure of valid reasoning• Engineering cycle is structure of valid reasoning about an

artifact– Rational decision making– Justification of a design

• Engineering process is the performance of engineering tasks– Management of finite resources and risk over time.– Finite resources (time, money, people, knowledge, capital, …)– Risk (undesirable events may happen)– Risk appetite (willingness of decision maker to run risks)


Outline




24

To design an artifactto improve a

problem context

To answer knowledgequestions about the artifact in

context

Problems & Artifactsto investigate

Knowledge,Design problems


Checklist questions about research context


1. Improvement goal?2. Knowledge goal?3. Current knowledge?

Engineering cycle Empirical cycle

17. Contribution to knowledge goal?18. Contribution to improvement goal?

4. … ….16. …


Research problem analysis4. Conceptual framework?5. Research questions?6. Population?

Research execution11. What happened?

Research & inference design7. Object of study?8. Treatment specification?9. Measurement specification?10. Inference?

Data analysis12. Data?13. Observations?14. Explanations?15. Generalizations?16. Answers?

Empirical cycle

New research problem

Design justification7. Object of study justification?8. Treatment specification justification?9. Measurement specification justification?10. Inference justification?

The research setup

• Case‐based research:– Objects of study are investigated one by one, in sequence– E.g. observational case studies, simulation, action research

• Sample‐based research:– Sample of objects of study are investigated as a whole– E.g. statistical surveys, randomized controlled experiments


Outline




• A theory is a belief that there is a pattern in phenomena


Theories in popular discourse

• In popular discourse:– A theory is an unvalidated speculation:

• “There is a secret government department that monitors all our email”

– An idealization not applicable to the real world:• “Merging two faculties reduces cost in theory, not in practice.” • “Traffic rules are fine in theory, but not on the street”.

– An opinion (value judgment) that may be denied by otheropinions (value judgments).

• “The Dutch lost the game because the players earn too much.”• “You should buy a Mac, then you will not lose your files anymore”


• A scientific theory is a theory that– Has survived tests against experience– Has survived criticism by critical peers


Sciences of the middle range

27 November 2014 © R.J. Wieringa 32

Generalization

Realism

Universalgeneralization

Existentialgeneralization

Casedescription

Idealized conditions Realistic conditions Conditions of practice

Basic sciencesPhysics, Chemistry, parts of Biology

Special sciences (about the earth):Biology, Psychology, Sociology, …Applied sciences:Astronomy, Geology, Meteorology, Political sciences, Management science, …Design sciences:Software engineering, Information systems, Computer sciences, Electrical engineering, Mechanical enghineering, ...

Case research:Engineering, Consultancy, Psychotherapy, Health care, Management, Politics, ...

Outline




The structure of scientific theories

1. Conceptual framework– E.g. The concepts of beamforming, of multi‐agent planning, of

location compliance

2. Generalizations stated in terms of these concepts, thatexpress beliefs that go beyond experienced phenomena.

3. Each generalization has a scope. The scope of a theory is the union of the scope of all of its generalizations.

– The generalizations of a theory usually all have the same scope.


Variable‐based theory• Conceptual framework defines constructs and variables.• Variables have data types and scales• Generalizations stated in terms of variables

– Case based (about class of similar cases)– Sample‐based (statistical generalization)

• Examples:– DOA descriptive theory: e.g. performance graphs– DOA causal theory: e.g. change in angle of incidence causes change in

phase difference– Introduction of agile development causes customer satisfaction to

increase– Programmer productivity correlates well with conscientiousness


Architectural theory

• Conceptual framework defines an architecture in terms of components and relationships

• Components have capabilities• Generalizations stated in terms of capabilities and interactions

of components• Examples:

– DOA mechanistic theory: e.g. input‐output relation is explained bystructure of the algorithm

– In agile development for SME, the SME does not put customer on‐site. SME resources are limited and focus is on business.

– Introduction of change control board reduces requirements creep.


The structure of design theories

1. Conceptual artifact framework– Usually a conceptual problem framework is assumed

2. Generalizations of the form Artifact X Context → Effects3. The scope of the theory is usually expressed in terms of

constraints on artifact design and assumptions about the context


Outline




The functions of scientific theories

• To analyze a conceptual structure• To describe phenomena (descriptive statistics, interpretation)• To explain phenomena• To predict phenomena (analytical or statistical generalizations

of a theory, or use models)• To design an artifact by which to treat a problem


The functions of scientific theories

• To analyze a conceptual structure• To describe phenomena (descriptive statistics, interpretation)• To explain phenomena• To predict phenomena (analytical or statistical generalizations

of a theory, or use models)• To design an artifact by which to treat a problem


Causal explanations (cause‐effect relation between variables)

• If Y has been caused by X, then Y changed because X changedearlier in a particular way

• Examples– Light is on because switch was turned– Cost increased because the organization had to perform additional

tasks• Causation may be nondeterministic

– Forward nondeterminism: X sometimes causes Y– Backward: Y is sometimes caused by X

• In the field, the causal influence of X on Y may be swamped bymany other causal influences. – Lab research versus field research


Mechanistic explanations (interactionsbetween components)

• If system phenomenon E was produced by the interaction of system components C1, …, Cn, then C1, …, Cn is called a mechanistic explanation of E.

• Examples– Light is on because it is connected by to electricity supply when switch

was turned on– Cost increased because new people had to be hired to perform

additional tasks

• May be nondeterministic• May be interfered with by other mechanisms in the field


• Single‐case mechanism experiments: Investigate underlyingmechanisms (interaction between components) in single cases– Test a single instance of the artifact in the lab/field– Technical action research: Use the artifact to solve real‐world problem

• Statistical difference‐making experiments: Investigate averageeffects (average difference between treating and not treating) in large samples


Small samples

Large samples

Population

More realistic conditions of practice

Larger generalizations

Idealized Practical

Statistical difference‐making experiments

Single‐case mechanismexperiments

Technical action research

Case‐based inference Sample‐based inferenceNo control overobject of study(observational study)

Observational case study Survey

Control over object of study (experimentalstudy)

Single‐case mechanismexperiment,Technical action research

Statistical difference‐making experiment


Summary of research designs and research goals


Research goalsResearch designs

Problem research / Evaluation research

Treatment survey Validation research

Survey To survey problemowners / implementations

To survey possibletreatments

To survey expert opinion

Observationalcase study

To study a problem/ Implementation

Single‐casemechanismexperiment

To diagnose a problem / Test animplementation in context

To test an artifactwithout context

To validate an artifactin context

Technical Action Research (TAR)

To use and validatean artifact in practice

Statistical difference‐making experiment

To comparetreatments on random samples

To comparetreatments on random samples

Take‐home

• Design problems vs knowledge questions• Middle‐range theories• Case‐based research and sample‐basedresearch

• Scaling up


Extra material

• Scientific inference


Four kinds of inference

• All inferences are fallible. • Validity is degree of support for the inference. Cannot be

total.


Case‐based inference


Sample‐based inference


Documents

Introduction to Design Science methodologyroelw/Introduction... · Research problemsin design science • “Design a DoA estimation system for satellite TV reception in a car.”