Impact Assessment of

Interactive Coupled Open

Innovation in Living Labs

dr. Dimitri SchuurmaniMinds Living Labs & iMinds – MICT - Ghent University

@DimiSchuurman

Dimitri.Schuurman@iminds.be

Dimitri SchuurmanSenior User Researcher Living Lab methodology

• Started working on Living Labs in 2005

• Initiator & manager iMinds SME Lab program

• Supported over 60 SMEs in user innovation

• PhD on Open & User Innovation in Living Labs

Living Lab definitionApproach to innovation

characterized by…

Multi-method

Real-life experimentation

Active user involvement (co-creation)

Multi-stakeholder (PPP-organization)

European Commission policy

support

European Paradox: exploration

(research) vs. exploitation (market

success)

2006: ‘big bang’ with the establishment

of

https://issuu.com/enoll/docs/enoll-print

https://issuu.com/enoll/docs/enoll-print

4

Different perspectives on user co-creation

Greenbaum (1993)

• the ethical or democratic perspective VISION

• the pragmatic or economic perspective SUSTAINABILITY

• the curiosity or theoretical perspective ACTION RESEARCH

Living Lab services for SMEs

SMEs struggle with the practical

implementation of Open Innovation

Living Labs complement R&D resources

SMEs lack

Two-way agility & flexibility

Openness for active user involvement

Willingness to iterate / pivot

https://issuu.com/enoll/docs/living_lab_services_for_business_su

https://issuu.com/enoll/docs/living_lab_services_for_business_su

6

iMinds SME Lab organization

User Research Business Modelling

Ghent University

6 FTEVUB

2 FTE

iMinds central

3 FTE

iMinds central

1 FTE

Business developmentPanel management

Segmented panel of end-users (>25.000)

Longitudinal

knowledge

accumulation

& retention

7

iMinds SME Lab organization

User Research Business Modelling

Ghent University

6 FTEVUB

2 FTE

iMinds central

3 FTE

iMinds central

1 FTE

Business development

Utilizers of the Living Lab: mostly SMEs (>50 projects)

Panel management

Segmented panel of end-users (>25.000)

Other iMinds researchers or organizations with

specific expertise

Longitudinal

knowledge

accumulation

& retention

Project-based

involvement

8

LL

INNOVATION

FRAMEWORK

LL USER

ACTIVITIES

KICK-OFF

LLAVA

WORKSHOP

SEGMENTING

(RECRUITING)

SURVEY

CONTEXTUAL

INQUIRY &

CO-CREATION

FIELD TEST

LARGE

SCALE

ADOPTION

SURVEY

SOTA +

EXPERT

INTERVIEW

EXPLORATION

REAL-LIFE OBSERVATION

PROBLEM – SOLUTION FIT

LLAVA

UPDATE

EXPERIMENTATION

REAL-LIFE TEST

PRODUCT – MARKET FIT

LLAVA

UPDATE

GO2MARKET

WORKSHOP

iMinds innovation project structure

CURRENT STATE FUTURE STATE

CUSTOMERSEGMENT

KEY MARKET TRENDS

COMMONNEED

CURRENT ALTERNATIVES

VALUEPROMISE

BARRIERS

DIGITAL SOLUTION

VALUE CAPTURE(WTP)

KEY PARTNERS

IMINDS LLAVA MATRIX - FRAMEWORK

Which customer segments / verticals to focus on? What are common characteristics?

How to interact with which stakeholders? Stakeholder or customer?

Customer resistance vs experience design (interface, process flow, services, …)

Real needs of customer segment + priorities?

What value (monetary and non-monetary) do I receive in turn? What price should I set (and how)?

What (measurable) impact will you create?

Customer experience gaps in current alternatives?

Technical, societal or business trends affecting the customer segment?

What are our strengths & key capabilities?

+50 methods for

learning and

discovery of social,

contextual and

functional

requirements. https://www.iminds.be/en/userinnovation

Five steps to impact assessment

Define the different levels of analysis of the to be studied phenomenon

Link the levels of analysis to relevant theoretical frameworks for your analysis

Operationalize these frameworks through measurable variables

Chose the data collection method & source

Analyze & iterate

LL

ORGANISATION

LL PROJECT

FRAMEWORK

LL USER

ACTIVITIES

USERSINFRA-

STRUCTUREACADEMIA

LLAVAWorkshop

LLAVAupdate

PRIVATE

SECTORDATA

EXPLORATION EXPERIMENTATION

The 3 levels of a Living Lab

LLAVAGO2M

PUBLIC

SECTOR

Schuurman, 2015

https://biblio.ugent.be/publication/5931264/file/5931265.pdf

https://biblio.ugent.be/publication/5931264/file/5931265.pdf

Living Lab organizations

400 ENoLL accredited Living Labs

Core of 170 currently active LLs

Variation in participating stakeholders &

thematic goals

Smart cities / Urban Living Labs

Rural development Living Labs

Technology / sector oriented Living Labs…

USERSINFRA-

STRUCTUREACADEMIA

PRIVATE

SECTORDATA

PUBLIC

SECTOR

https://issuu.com/enoll/docs/enoll-print

https://issuu.com/enoll/docs/enoll-print

Open

Innovation

Organizations benefit by opening up their

innovation processes to external actors in

order to exchange knowledge &

technologiesHenry Chesbrough (2000’s)

Open Innovation – Organization perspectiveMain idea: “Knowledge and technology transfers can and should be actively managed”

Challenges: Managing and balancing open and closed innovation

OI processes Exploration Exploitation Retention

Get new knowledge in Get internal knowledge out Reuse knowledge base

°2004

2.500 users with PDAs

100 Hotspots

ICT

8 consortium actors

No external projects

°2010

2.015 Panel members

3DTVs, tablets, sensors

Smart grids, cities & media

5 consortium actors

3 external projects

°2010

>2.015 Panel members

>7.000 thematic dataset

(AV) media

5 consortium actors

>15 external projects

°2010

115 connected homes

FttH network, tablets, MiniPCs

Multimedia, eCare & gaming

12 consortium actors

3 external projects

A decade of Flemish Living Lab constellations

Impact: linking goal with outcome

Total success % Exploration success % Exploitation success %Providers 8 out of 16 50% / / 1 out of 3 33%Researchers 4 out of 5 80% / / 0 out of 1 0%Utilizers 18 out of 26 72% 11 out of 12 92% 3 out of 7 43%Total 15 out of 23 65% 11 out of 12 92% 4 out of 11 36%

Success rate for SMEs

Total succ % Exploration success % Exploitation succ %

Enablers 5 out of 7 71% 3 out of 5 60% 2 out of 2 100%Providers 8 out of 16 50% 1 out of 4 25% 7 out of 12 58%Researchers 4 out of 5 80% / / 4 out of 5 80%Users 4 out of 4 100% 4 out of 4 100% / /Utilizers 18 out of 26 72% 14 out of 17 82% 4 out of 9 44%Total 39 out of 58 67% 22 out of 30 73% 17 out of 28 61%

Success rate per actor

Data collection: Interviews with Living Lab managers + document analysis

Impact: linking goal with outcome

Success %

FLELLAP 6 out of 11 55%

LeYLab 11 out of 18 61%

Mediatuin 10 out of 15 67%

iMinds Living Labs 12 out of 14 86%

Total 39 out of 58 67%

Success rate per Living Lab

Qualitative enrichment

Living Lab projects

Project methodology, Living Lab specific

Tailored towards the utilizers of the Living

Lab services (e.g. SMEs)

Focus on multi-method, iterative

development and real-life experimentation

LLAVAWorkshop

LLAVAupdate

EXPLORATION EXPERIMENTATIONLLAVAGO2M

Living Lab project outcomes

Project Yes No

During 7 - 54% 6 - 43%

After 4 - 31% 3 - 21%

None 2 - 15% 5 - 36%

On the

market4 - 31% 6 - 43%

Pipeline 6 - 46% 2 - 14%

Reoriented 3 - 23% 6 - 43%

http://timreview.ca/article/956

Data collection: Post hoc interviews with project instigators

+ document analysis

21

Abort19%

Reboot7%

Adapt40%

Launch35%

The product is available

on the market.

The product is still under development

but expected to go to market soon.

Initial product is aborted, but a new

product idea roadmap is initiated.

Product development is discontinued or

will not go to market in its current form.

Own impact assessment instrument (N=35)

Primary added value: Input for innovation development

User contribution: evaluation (97%), incremental improvement (97%) & substantial

improvement (77%)

Data collection: Pre & post hoc interviews with project instigators

22

Independent impact assessment study14 SMEs that engaged in Living Lab projects

Economic benefits

• Investment 3,8 mio €

• Employment 79 FTEs extra

• Additional revenue 13,1 mio €

Added value for NPD process

• 93% indicates project results were

useful or extremely useful

• 50% used the results to improve the

innovation before market launch

• Main reasons to engage in LL-project:

• User research

• Structured methodology

• User & bus modeling combo

Living Lab user activities

Methods & tools that involve end-users

Focus on active user involvement & co-

creation

Different toolsets in different Living Labs

User Centered Design & Design Thinking

(Lead) User Innovation

Voice-of-the-Customer techniques

+50 methods for

learning and

discovery of social,

contextual and

functional

requirements.

https://www.iminds.be/en/userinnovation

https://www.iminds.be/en/userinnovation

User

Innovation

Users can and will contribute

to innovation processes or

even innovate themselves

Eric von Hippel (1970’s)

25

USER INNOVATION

Voice-of-the-Customer User co-creation Lead User methods

Innovation FOR usersInnovation WITH

usersInnovation BY users

PASSIVE ACTIVE

Who? Different types of users

Outcome? Radical / incremental innovation

Problem? How to facilitate and structure

co-creation for the total duration of the

project?

How? Iterative combination of voice-of-the-customer methods with

generative methods and techniques

26

User participation in co-creationData collection: Survey with panel members & interviews with alpha users

27

Define success

Greenbaum (1993)

• the ethical or democratic perspective active user involvement

• the pragmatic or economic perspective improved innovation outcomes

• the curiosity or theoretical perspective multi-stakeholder collaboration

Impact assessment model

Based on document

analysis & interviews with

Living Lab managers

Based on interviews with

project instigators &

document analysis

Based on surveys &

interviews with users

Level of

analysisIndicators Perspective N

Macro – Living

Lab organization

Open Innovation

processesTheoretical

4 LL

organisations

Meso – Living

Lab project

Innovation process

outcomesEconomic

21 LL

projects

Micro – Living

Lab user activities

User Innovation

contributionsDemocratic

107 research

activities

Current instruments

Level of analysis Indicators Perspective

Macro – Living

Lab organization

Open Innovation

processesTheoretical

Meso – Living

Lab project

Innovation process

outcomesEconomic

Micro – Living

Lab user activities

User Innovation

contributionsDemocratic

None!

Pre & post survey project

instigator / environmental

scan

Short post survey at co-

creation sessions

Separate sessions &

surveys with panel

members

Five steps to impact assessment

Define the different levels of analysis of the to be studied phenomenon

Link the levels of analysis to relevant theoretical frameworks for your analysis

Operationalize these frameworks through measurable variables

Chose the data collection method & source

Analyze & iterate

Issues & challenges

Action research

Timeframe for data collection & analysis

Experimental data

Relate specific contributions to specific users and/or research activities

dimitri.schuurman@iminds.be

@DimiSchuurman