Towards Citizen Co-created Public Service Apps

Diego López-de-Ipiña1, Mikel Emaldi1, Unai Aguilera1, Jorge Pérez2

1 DeustoTech-Deusto Foundation, University of Deusto, Av. Universidades 24, 48007 Bilbao, SPAIN

2TECNALIA, eServices, Spain {dipina, m.emaldi, unai.aguilera}@deusto.es,

jorge.perez@tecnalia.com

Abstract. This paper describes the WeLive framework, a set of tools to enable co-created urban apps by means of bringing together Open Innovation, Open Data and Open Services paradigms. It proposes a more holistic involvement of stakeholders across service ideation, creation and exploitation. The WeLive co-creation process applied to three new urban apps in the city of Bilbao is de-scribed. The two-phase evaluation methodology designed and the evaluation re-sults of pre-pilot sub-phase are also presented. As a result, an early user experi-ence evaluation for WeLive has been obtained.

Keywords. Open Government, Open Data, Open Services, Open Innovation

1 Introduction

The 2010 edition of the EU eGovernment Benchmark Report [1] states that currently public services are built following an administration-centric approach, driving to a low usage, rather than according to the citizens´ need (user-centric approach). There-fore, there is a clear need to move towards a more open model of design, production and delivery of public services involving citizens, entrepreneurs and civil society.

Public administrations are facing key socioeconomic challenges such as demo-graphic change, employment, mobility, security, environment and many others. The squeeze on public finances has created renewed momentum for the modernisation of public administration. The fact is that in the EU, according to data of 2013 [2], public expenditure accounted for almost 50% of GDP and the public sector represents about 17% of total employment. Besides, citizens’ expectations, in terms of burden reduc-tion, efficiency, and personalization, are growing. Citizens want to transit from being mere consumers of public services to providers of those services, i.e. prosumers of the open government ecosystem.

ICT-enabled Open and Collaborative Government is the recipe to deliver "more from less", i.e. meeting public needs in times of tighter budgets, improving the busi-ness environment by providing better services to business and citizens, and adapting service provision to the needs of a more digital economy [3].

The WeLive project is devised to transform the current e-government approach by facilitating a more open model of design, production and delivery of public services

based on the collaboration among public administrations (PAs), citizens and entrepre-neurs. WeLive comprises a novel ecosystem of tools built on the Open Data, Open Services and Open Innovation paradigms, easily deployable in different public admin-istrations. It aims to promote co-innovation and co-creation of personalised public services through a bottom-up approach that leverages active participation of different stakeholders. As shown in Fig. 1, WeLive’s Open & Collaborative Government ICT infrastructure resembles an assembly line for e-government services. It offers tools to transform the needs into ideas, then tools to select the best ideas and create the build-ing blocks necessary to build the envisioned solutions, and finally a way to compose the building blocks into mass market apps which can be exploited through a market-place.

This paper has the following structure. Section 2 reviews related work. Section 3 describes the WeLive platform and its key components. Section 4 outlines the WeLive co-creation approach. Section 5 describes the evaluation methodology de-signed and its application to the pilot in Bilbao. Finally, section 6 draws some conclu-sions.

Fig. 1. WeLive concept: IDEAS >> APPLICATIONS >> MARKETPLACE

2 Related work

WeLive deals with the following topics: 1) Open Data, 2) Open Services and 3) Open Innovation and Co-creation. To enable a user-centric collaborative public service ecosystem, new types of infrastructure are needed where public value is created by the ability to share, interact and collaborate between actors. Novel processes and ap-proaches are required with a decentralised, cross-government and multi-actor archi-tecture, coupled with the integration of Big Data and the role of social tools [4]. Po-tential problems related to security, privacy and data protection issues have also to be tackled. An open governance framework, as intended by WeLive, also requires in-teroperability (both at the organisational and technological level), open standards, adoption of Linked Data principles and Cloud Computing. In terms of web evolution, a solution like WeLive has to embrace 3.0 development, i.e. machine integration of data. Research demonstrates the potential of mobile apps to transform governments

and to provide access to public information; m-Government is therefore emerging as the next big wave for ICT use in public sector [5].

Open Innovation [6] provides a framework for involving actual customers in the service and product innovation process. It is a co-creational process that facilitates stakeholder engagement, where companies and governments should use external ideas to develop their services in a collaborative Public Private Partnership setting. With the increased focus on sustainability and development of Smart Cities, a more inclusive public administration is enabled, i.e. one that incorporates citizens into the planning of development activities through different channels enabling their engagement on vari-ous matters. This has resulted in Living Labs [7], e.g. cities used as places where new services can be deployed and tested in a real environment, capturing the feedback and contributions crowdsourced by the citizens.

Since the first public administrations started sharing their data as Open Data, the idea of Open Government has been disseminated around the world rapidly. As de-scribed by the Open Data Barometer [8], Europe leads the region ranking, with widely known initiatives like data.gov.uk (UK) or opengov.se (Sweden). However, most of countries which have started adopting Open Government policies have serious lacks on exploiting the potential of Open Data. Many of them are focused only on imple-menting open data portals, placing little efforts on bringing open data closer to entre-preneurs and citizens through APIs, consumable by application developers, which is aimed by WeLive.

3 The WeLive platform

WeLive encompasses a suite of tools that provides companies, governments, re-searchers and citizens with the capability to collaborate in order to promote innova-tion. WeLive supports innovation from the co-creation of innovative ideas (through the Open Innovation Area component) and their implementation, to the publication of artefacts (i.e. datasets, building blocks (BBs) and public service applications) in the WeLive Marketplace. Furthermore, developers who want to create services or build-ing blocks for the WeLive ecosystem can freely choose diverse technologies (such as programming language, operating system, third party libraries) which they find rea-sonable, as long as they host the service/BB by themselves. Still, WeLive also pro-vides hosting environments for those who do not want to set up and maintain their own servers.

The WeLive platform architecture is structured into four layers, see Fig. 2, where a central component, WeLive Controller, orchestrates requests issued through the WeLive web UI1 or the WeLive RESTful API2, see Fig. 3, to other components:

1. The Open Innovation layer is dedicated to boost collaborative research and devel-opment as a mechanism to fuel innovative discovery of novel public services.

1 https://dev.welive.eu/ 2 https://dev.welive.eu/dev/swagger/

2. The Open Data Layer handles data managed by the public administration which may have been contributed directly by the administration but also provided by the end-users through their smartphones’ apps or extracted from social networks.

3. The Open Service Layer helps public administrations, companies and citizens to realize new added value services on top of the Open Data layer. It includes the Marketplace component where WeLive artefacts are published.

4. The Intelligence Layer: Personalization & Analytics provides mechanisms to cus-tomize public services using users’ personal information. Personalized services use a variety of data related to the users (memorized in the Citizen Data Vault compo-nent) to adapt their behaviour and to provide significant user benefits thanks to services that better support them (recommended by the Decision Engine compo-nent). At the same time, this layer is used by the WeLive stakeholders to monitor and evaluate the available artefacts (both in the form of ideas and as actual applica-tions) in the WeLive ecosystem through the Analytics Dashboard component.

Fig. 2. WeLive platform architecture.

3.1 Open Innovation Layer

This layer is embodied by the Open Innovation Area component. Such component is a social co-creation environment where needs, ideas and possible "solutions" can be matched and asked to Public Administrations for implementation. This is the place where requests meet possible offers. Needs are made public and so highlighted to the community. Several ideas are suggested to satisfy a specific need (e.g. coming from citizens or directly from PAs). It offers tools for: a) eliciting, analysing and improving ideas; b) to vote and select the ‘best’ ideas for a need, c) to allow companies to offer technical solutions to selected ideas and be funded by interested citizens or the P.A.

It implements the following procedural steps to carry out Open Innovation within a city: a) suggest or report common needs to the Authorities, for the “development” of the territory; b) co-define, through social and collaboration features, new ideas; c) evaluate and select the best ideas through voting and commenting; d) monitor the status of ideas consisting of co-experience, co-definition, co-development and co-delivery phases: e) contribute to the idea implementation through the interaction with the Visual composer, a future WeLive component which will enable non-technical people to mash-up building blocks; and f) associate ideas with artefacts published into the Marketplace, to track the relationship between ideas and datasets, BBs and public services.

Fig. 3. WeLive Web front-end.

3.2 Open Data Layer

This layer is responsible for managing Broad Data [9] gathered within a city, coming from different heterogeneous sources such as Open Government Data repositories, user-supplied data through social networks or apps, public private sector data or end-user personal data (which can be used to filter and recommend the contents and ser-vices most relevant to them). For that, it provides the Open Data Stack and the Citizen Data Vault components. The former serves to address the whole lifecycle of broad data management: data discovery, capture, curation, storage, linkage, publication, exploitation and even visualization. The latter one is used to keep a safety vault where the user’s personal data, preferences and profile details are accessed only under user control. The ODS functionality consists of the following internal modules:

• Data source management: provides the functionality to perform CRUD operations with heterogeneous datasets and its associated resources. It provides a user inter-face to facilitate the management of this data.

• Rating system: manages the ratings of datasets and its associated resources provid-ed by Open Data Stack users. They can provide a rating for a dataset and the sys-tem will compute the aggregated rating for all the users.

• Validator: this module checks the data associated with a dataset/resource and vali-dates it applying a set of rules, e.g. JSON schema, which can be defined and ex-tended by the users. The results of the validation process is shown through the data source management component to provide a seamless experience to the users.

• Harvester: the harvesting module manages all the information and functionality related with automatic data extraction from external sources. User can configure the existing harvesters, i.e. web scrapper and twitter harvester, or create new ones to extract and process data from non-structured data sources.

• Query Mapper: this module provides functionality to map external data sources and perform queries and updates to them using a common query language SQL. Supported formats are: JSON, CSV, Relational databases and SPARQL. This component was created as part of the IES Cities European project [10]. The in-teroperability between other WeLive platform components and the ODS is facili-tated by it.

• API: this module provides a programmatic access interface to all the functionality of the ODS, which is useful for the integration of this component with other WeLive project’s component and its usage by developers when creating applica-tions.

3.3 Open Services Layer

This layer is responsible for managing the main public service artefacts handled by WeLive, namely public service apps –HTML5 Web apps or mobile apps executable in diverse mobile devices by final users, building blocks – represent a digital capabil-ity of a smart city made available to developers through a REST API, and dataset – data of interest to create a new building block or app. This layer comprises the com-ponents:

• WeLive Marketplace. It represents the repository where to look at in order to iden-tify available apps/building blocks/datasets, i.e. WeLive artefacts. It is the core component to exploit the economic potential brought forward by WeLive. There, WeLive artefacts can be browsed, selected, purchased (not implemented yet) by different stakeholders following SaaS and DaaS [10] approaches.

• Hosting environments. They host WeLive artefacts’ business logic back-end. Such hosting might be done in one of the environments provided by WeLive or, alterna-tively, at a third party, Cloud-based IaaS or PaaS, as long as artefacts hosted there are WeLive Open Services framework compliant. The project currently offers: a) a CloudFoundry-based hosting environment as a general-purpose solution to host web apps in the cloud; and b) a hosting environment operated within Cloud’N’Sci.fi marketplace for hosting computationally complex and/or commer-cialized BBs.

• WeLive Player. An app (see Fig. 4 leftmost screenshot) to access the city service ecosystem. It permits to search and discover the public service apps of the city, fil-tered through user’s context, profile, and preferences.

Artefacts deployed in a hosting environments must provide standard descriptions describing both general-purpose information (e.g., providers and authors, license info, classification) and technical information (e.g., the supported protocols and formats, security constraints). The specification, which relies on the Linked USDL [12] nota-tion, a formal semantic specification language, aims at representing the following elements of the building block life-cycle: a) BB definition (name, description, pub-lisher, etc.); b) BB invocation and execution (interaction points); c) BB access poli-cies (security profile); d) BB authoring, ownership, and licensing aspects; and e) BB service level agreements. The aim of this specification is to capture the BB character-istics that may be required when: a) users (BB clients, citizens, developers) search for relevant components to build new applications upon them; and b) during the applica-tion execution, when a BB has to be accessed and invoked providing the unambigu-ous definition of access policies and invocation protocols. Besides, the programmable resources defining the BB metadata should be exposed according to well-defined interfaces (URLs). For example: an HTTP GET issued to {build-ing_block}/spec/list should return the list of specification formats it sup-ports (it should include at least xwadl and usdl), or the following URLs gives access to the WADL [13] descriptions of the BB: {building_block}/spec/xwadl.

Besides, the Open Service Layer provides a set of key infrastructure components, referred to as Core Building Blocks. They expose operations to be used by both by the platform itself and by new building blocks. The most remarkable ones are:

• Authentication and Authorization Control (AAC) BB provides the functionality of user authentication and authorization across apps and tools in WeLive platform.

• Logging BB allows for tracing information regarding the platform-level and appli-cation-level events.

4 WeLive co-creation approach

The service co-creation and innovation approach in WeLive is based on the "Quadru-ple Helix" paradigm [14], where the fourth helix is the citizen, who closely collabo-rates with Industry, Government and Academia to co-create value added public ser-vices.

Through WeLive platform, stakeholders participate by contributing with new ser-vices, building blocks and dataset requests, by giving place to new data and apps and even by co-funding novel public app ideas. Stakeholders in a city consume the gener-ated apps publicly available in the WeLive Marketplace and actively contribute and collaborate co-creating public services of high economic and social interest for a city.

During the first half of the project, first 18 months, the following co-creation pro-cess has been applied. Online questionnaires addressed to citizens, public servants and SMEs staff have been issued at each of the three cities and one region participating in

the project to understand what new urban apps are most requested at each of them. In parallel, an analysis of each participating city strategy and the status of its IT and open data infrastructure was carried out. The analysis of the information gathered through these two channels was followed by the celebration of several focus group workshops where face to face contact with different types of stakeholders took place. In such workshops, the discussion was guided by selecting a couple of scenarios per city, based on the opinions of stakeholders and the city strategy analysis.

4.1 Bilbao’s co-creation process experience

With regards to Bilbao’s case, in first place, questionnaires among different stake-holders (citizens, companies and public administrations) were distributed by e-mail. The following numbers of people were contacted: 271 citizens, 43 companies and 10 public administration representatives. Besides, the city strategy documents were ex-plored. Looking at the questionnaire results and the city strategy, the topics of traffic, culture, health and tourism, were identified as most relevant for stakeholders.

As a second step, two motivating scenarios, based on the most relevant topics, were used as discussion points for the three focus groups sessions celebrated: a) “Col-laboration in the Neighbourhood and Participation in municipal decision” and b) “Collaborating on a more Efficient and Transparent City”. A workshop for each stakeholder group was celebrated where people participating in the initial survey and other representatives were invited. For each session, people were divided in groups of 4-5 people, each of them reflecting in one of the scenarios. At each session, the fol-lowing steps were carried out: a) Inspiration phase –project partners presented the project idea and the co-ideation approach of WeLive; b) Brainstorming about ideas associated to the scenarios assigned to each group; c) Voting for ideas by all groups; d) Specific service or use case definition; e) Presentation of services conceived; and f) Voting for services.

From these focus group sessions, a selection of six potential new urban apps for the city of Bilbao were identified, three for each of the two scenarios.

4.2 Population of WeLive environment for Bilbao

Based on the feasibility analysis of each of the candidate services, given that some relied on the assumption of existence of certain datasets in the public domain, three services have been implemented so far. In two of the selected services, i.e. Bilboener-gy and Public Spent Dashboard, the datasets were not ready yet. The other apps re-quired datasets existing or easily created, since they would be populated by the users themselves. Hence, WeLive instance for Bilbao was populated with the following artefacts:

1. Datasets. They were imported both from Bilbao Open Data3 and Open Data Euskadi4. A total of 165 datasets are currently available in the ODS component as seen in https://dev.welive.eu/ods/organization/bilbao-city-council.

2. Building blocks. A range of BBs, both specific to the apps and generic across the whole project were created as referred later.

3. Apps. Three new apps were developed, published in WeLive marketplace and in Google Play, namely Bilbozkatu, BilON and Auzonet. ─ Bilbozkatu - a proposal box for citizens to make improvement suggestions for

their neighbourhoods, which others can support or not. Its main features are: create proposal, vote in favour or against a proposal, comment and rate pro-posals and see proposals geo-located on a map. Internally, it makes use of the following BBs: Feedback BB, In-app questionnaire, AAC, Logging. A new da-taset has been created within ODS to support the voting campaigns. Information about neighbourhoods in Bilbao is retrieved from Bilbao Open Data.

─ BilbON – app with points of interest (POI) in Bilbao classified in categories. Its main features are: a) display categories of POIs on a map, b) search for POIs by different criteria, c) see details of a POI or d) create new POIs. Internally it makes use of the following building blocks: In-App questionnaire, AAC, Log-ging. A new dataset was created within ODS to support the voting campaigns, whilst POIs datasets are obtained from Open Data Euskadi. Fig. 4 shows snap-shots of this app.

─ Auzonet – app to share goods/items between neighbours. Its main features are: a) offer a good or item, b) demand a good or item, c) rate other users and d) chat with people who offers or demands an item. Internally, it makes use of the fol-lowing BBs: Rating BB, In-App questionnaire, AAC, Logging. A new dataset has been created within ODS to support the exchange of articles among neigh-bours and rating of those exchanges.

5 WeLive evaluation methodology

WeLive project is being executed in two phases. In phase I, only the Open Innovation Area, Open Data Stack and WeLive Marketplace components have been made public-ly available. This explains why the first developed apps have been the result of an open participation process, rather than relying exclusively on the platform. Fig. 5 shows the timeline for the first phase of pilots, being executed. As can be seen, the execution, monitoring and evaluation of the pilots has been divided in two sub-phases: pre-pilot sub-phase – addressed to a controlled group of users that assess the platform and apps – and pilot execution sub-phase – addressed to the overall public.

3 http://www.bilbao.net/opendata/es/inicio 4 http://opendata.euskadi.eus/w79-home/es

Fig. 4. BilON co-created service launched through WeLive Player

5.1 Evaluation tools and KPIs in WeLive

A set of tools have been created in order to evaluate WeLive tools and apps and measure their take-up along the pilots’ execution. Firstly, data logs from apps are collected to give information about users’ activity on the WeLive platform and their usage of different apps are automatically submitted to the Logging Core BB. For in-stance, an ArtefactPublished event notifies when an app, BB or dataset is pub-lished.

Fig. 5. 1st pilot phase evaluation sub-phases, aim, testing groups and tools used.

Secondly, a short intuitive one screen in-app questionnaire, including 5 questions and 1 comment box, which can be launched from each WeLive app has been created. The objective of the in-app questionnaire is to gather information about: (1) intui-tiveness of an application; (2) usefulness of an application; and (3) acceptability of an application. An exemplary question is: Was this app useful for you?

Thirdly, a more extensive on-line questionnaire has been created to gather infor-mation about user experience and acceptability of WeLive concept and applications related to: (1) ethical, (2) legal, (3) economical, and (4) transparency aspects.

The generated data logs and the answers collected from in-app and online ques-tionnaires serve to feed with values a range of Key Performance Indicators (KPIs)

defined across the project. A set of generic KPIs associated to the different compo-nents and purposes of WeLive have been defined. For example: a) KPI1.2: Number of public services apps, building blocks and dataset ideas (per pilot): 64, or b) KPI3.1: Number of building blocks created with the help of the Open Service framework and Open Data Toolset (for all pilots): 9. Likewise, for each of the apps, a range of KPIs was defined to be able to assess the success of their deployment. For example, for BilbON app some of the KPIs are: a) KPI.BIO.12: ' Number of times the app is start-ed: > 150 or b) KPI.BIO.14 ' Number of new POIs added: 80% of users create a POI. Observe that each defined KPI is associated to a priori defined target value.

Fig. 6. Preliminary User Evaluation of WeLive solution.

5.2 Pre-pilot sub-phase execution and evaluation

In this completed sub-phase, the technical reliability of the solution has been assessed and the population of WeLive at the four pilot sites has taken place. The aim is to ensure that WeLive platform tools and city or region apps are robust and functional enough to be made publicly available at each pilot site.

WeLive tools and apps were tested in the following aspects: usability and friendli-ness, proper behaviour (prosumer and social interaction) and in-app questionnaires launching and proper log generation. The procedure applied in the alpha testing was: a) Alpha testers’ identification & groups’ creation; b) Alpha testers training work-shop. Each country task force tests the city public services, WeLive player and framework components (Controller, OIA, ODS, CDV, MKT & ADS); c) Apps/framework components testing including: usability and functionality issues identification, In-app questionnaires filling (only for the apps) and logs generation checking; d) Alpha script questionnaire filling (per participant) available at: http://bit.ly/2b54FwX, and e) Summary of the main issues identified and recording in Redmine tracking system used in project.

Besides, the project experience of use (UX) has been assessed. UX according to the international standard ISO 9241-210 consists of the user’s emotions, perceptions, preferences, physical and psychological responses, behaviour, and responses that result before, during or after use of a product. To get a first impression, the User Ex-perience Questionnaire (UEQ) [15] has been employed. UEQ aims to catch the im-mediate impression of a user towards a product. The UEQ contains 6 scales with 26 items, each item valued -3 to 3, e.g. attractive -3…-2…1…0…1…2…3 unattractive:

-2

-1

0

1

2

• Attractiveness: Overall impression of product. Do users like or dislike the product? • Perspicuity: Is it easy to get familiar or learn how to use the product? • Efficiency: Can users solve their tasks without unnecessary effort? • Dependability: Does the user feel in control of the interaction? • Stimulation: Is it exciting and motivating to use the product?

Opinions from 2 public servants and 7 SMEs were collected after celebrating a workshop with each of the collectives informing about WeLive tools and apps in the city of Bilbao. The obtained results are close to 0.8 value in all aspects except novel-ty, which according to the UEQ authors is already considered as a positive evaluation. Continuous improvements of apps and platform during the pilot execution sub-phase should enhance the results by the end of the first pilot. Moreover, audience less famil-iar with Open Government will be approached which should enhance the novelty rating.

6 Conclusion and Further Work

This paper has presented the WeLive platform tools and its co-creation approach ap-plied in order to given place to the first generation of WeLive compliant apps. The evaluation methodology designed to be applied in two phase pilots has also been out-lined. The first preliminary evaluation results, obtained during the first iteration of trials, pre-pilot sub-phase, associated to the current version of the platform and the first available apps in the city of Bilbao have been presented.

Currently the project is embarked in actual pilot phase I execution sub-phase where actual measurements about the overall acceptability of the WeLive solution in terms of the project wide defined KPIs will be obtained. Future work will gain insights about the actual acceptance of the solution within 4 distinct European urban contexts.

Acknowledgements

This work has been carried out within WeLive project H2020-INSO-2014, 645845.

References

1. European Commission, Directorate General for Communications Networks, Content and Technology, Public Services Online: ‘Digital by Default or by Detour?’ Assessing User Centric eGovernment performance in Europe. In: eGovernment Benchmark 2012, https://ec.europa.eu/digital-agenda/sites/digital-agenda/, (2013)

2. Annual Growth Survey 2013: Communication from the Commission, http://ec.europa.eu/europe2020/pdf/ags2013_en.pdf, 28.11, (2012)

3. Annual Growth Survey 2014: Communication from the Commission, http://ec.europa.eu/europe2020/pdf/2014/ags2014_en.pdf, 13.11, (2014)

4. Cabinet Office, UK: Public Service Mutuals - The Next Steps, A Mutuals Taskforce Re-port, June 25 2012

5. ITU, OECD: M-Government - Mobile Technologies for Responsive Governments and Connected Societies. In: http://www.itu.int/pub/D-STR-GOV.M_GOV-2011, 2011

6. Baldwin, C., von Hippel, E.: Modeling a Paradigm Shift: From Producer Innovation to Us-er and Open Collaborative Innovation, 2009

7. Almirall, E., Wareham, J. (2011): Living Labs: Arbiters of Mid- and Ground- Level Inno-vation. In: Technology Analysis and Strategic Management, 23(1), pp. 87-102, (2011)

8. Open Data Institute: Open Data Barometer – 2013. In: Global Report, http://www.opendataresearch.org/dl/odb2013/Open-Data-Barometer-2013-Global-Report.pdf, (2013)

9. Roden D.: Introducing Broad Data, http://www.domo.com/blog/2013/03/introducing-broad-data/, (2013)

10. Aguilera U., López-de-Ipiña D., and Pérez-Velasco J.: Collaboration-Centred Cities through Urban Apps Based on Open and User-Generated Data. In: Sensors. 16-7, 1022 (2016)

11. Dyche, J.: Data-as-a-service, explained and defined". In: SearchDataManagement.com. Retrieved October 24, (2010)

12. Linked USDL, https://linked-usdl.org/, (2016) 13. WADL Specification, https://wadl.java.net/, (2016) 14. European Commission, Digital Single Market. Open Innovation 2.0. In:

https://ec.europa.eu/digital-single-market/en/open-innovation-20, 2016 15. Rauschenberger, M., Schrepp, M., Cota, M.P., Olschner, S. & Thomaschewski, J.: Effi-

cient measurement of the user experience of interactive products - How to use the User Experence Questionnaire (UEQ). Example: Spanish Language Version. In: International Journal of Interactive Multimedia and Artificial Intelligence, 2(1), pp 39- 45. (2013)