NIC Linked Data: the OHIO project

NIC

Linked Data

The OHIO (Open Health Innovation Ontology) Project

Michael Wilkinson

NIC

Michael Wilkinson, NIC Semantic Web Page 1 of 23 09 April 2023

1.Executive SummaryThe emergence of the semantic web provides a significant opportunity to improve the speed and quality of decision-making in the area of health technology innovations.

This Business Case documents how the NIC will continue to develop its semantic web capability in order to deliver significant benefits to key stakeholder groups. This document starts with a detailed analysis of the current market conditions. It then outlines NIC progress to date as well as projected deliverables for June and October. This document then considers potential benefits to the NHS, and the approach to evaluation as a function of ongoing improvement. This document also presents a risk analysis, including a discussion on how such risk will be mitigated – thus ensuring that the potential benefits are likely to be realized.

The projected benefits arising from this investment, which will be measured, are expected to exceed the cost of this initial investment by a significant margin.

The NIC Linked Data Initiative will contribute to a number of key deliverables that the DH is expecting from the NIC, including contribution to improved:

o decision-making by clinicians, managers and commissioners in the NHS,

o support for healthcare industries,

o joined-up working with other government departments.


2. Market AnalysisThe NIC is in a prime position to take the leading role in delivering compelling semantically-enhanced applications to deliver value in the health innovation space. The challenge for the NIC is to assure delivery and real benefits in a timely fashion. This paper describes, at a high level, the business case.

o Needs addressed by the Semantic Web

o Emerging Opportunities in Healthcare

o NIC Progress to Date

o Next Steps for the NIC, with specified time frame, and cost

The NIC Website (www.nic.nhs.uk)


The Opportunity

The CBI (Feb, 2010) recently reported that the ‘NHS is data rich and information poor’, and noted that there is a pressing need to improve access to high-quality, utilisation-focused information in order to improve efficiency, effectiveness and quality in the NHS

The data-rich, information-poor challenge is not restricted to the NHS. As reported by The Economist (Feb, 2010), the world contains an unimaginably vast amount of digital information which is getting ever vaster ever more rapidly. This makes it possible to do many things that previously could not be done: spot business trends, prevent diseases, combat crime and so on. Managed well – and this is a pressing challenge - data can be used to unlock new sources of economic value, provide fresh insights into science and hold governments to account.

Governments around the world are freeing-up data to promote open innovation and open government. The US has taken the lead in increasing public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government through the creation of www.data.gov, which provides a limited portion of the rich variety of Federal datasets presently available. It has also invited people to actively participate in shaping the future of data.gov by suggesting additional datasets and site enhancements to provide seamless access and use of Federal data. This gives people a single place to go for all anonymous Government datasets to make it easier to reuse and innovate with Government data.

Informed from the US experience, the British Government recently launched data.gov.uk. This site seeks to give a way into the wealth of government data in a way that is easy to find, license and re-use by publishing data as RDF. Within months of its launch, the site contains nearly 3,000 freely available datasets from across government, 364 of which are from the Department of Health.

Making sense of all this data

Information has become super-abundant.

With so much information on the internet already, access to quality, context-specific information delivered to people when, where and how they want it is a challenge for all sectors. To meet this challenge, the business of information management has steadily grown in recent years. Estimated to be worth more than $100 billion now, this sector is growing at almost 10% a year, roughly twice as fast as the software business as a whole.

A key enabler for this development is the Internet itself, which is currently at the cusp of a transformation, moving from Web2 - social networking sites - to Web3 - semantic applications The Semantic web enables data to be linked to and expressed in meaningful ways, delivering a wide-range of benefits to end-users. The critical path to success is to understand end-user needs, then to develop applications that meet such needs. The global race to build semantic applications started in earnest at the WWW Conference in April 2009, where the full potential of linked data was demonstrated.

In April 2009, the NIC joined this race.

The prize to those who succeed at building the first set of compelling semantic applications is significant, given how important speed and access to quality information is to clinical decision-making, investment, and performance in the global economy.


http://blog.semantic-web.at/2009/04/22/tim-berners-lee-we-need-data-on-the-web-to-work-better-together/

http://en.wikipedia.org/wiki/Semantic_Web

http://data.gov.uk/home

http://www.economist.com/opinion/displayStory.cfm?story_id=15579717&source=most_read

http://www.cbi.org.uk/ndbs/press.nsf/0363c1f07c6ca12a8025671c00381cc7/1b39818b14156fa9802576be0034d281/$FILE/CBI%20Best%20of%20Health.pdf

Having produced compelling applications to empower key NHS Stakeholder groups, the NIC will be well-positioned in its delivery on a number of key DH policy initiatives, including:

o DH Innovation Procurement Plan (DH: 2009); o Office of Live Science: Delivering the Blueprint (DBIS & DH, January 2010), and o Operational Efficiency Programme: Final Report (HM Treasury: April, 2009).

Components of the Linked Data System

There are a number of key components of the Linked Data system that are important to ensuring that the system functions optimally. Described in more detail, below, these components include: ontology, triple-store, data, and applications.

Ontology

An ontology is is the philosophical study of the nature of being, existence or reality in general, as well as the basic categories of being and their relations. In Computer science an ontology is a formal representation of the knowledge by a set of concepts within a domain and the relationships between those concepts. It is used to reason about the properties of that domain, and may be used to describe the domain. An ontology provides a shared vocabulary, which can be used to model a domain – that is, the type of objects and/or concepts that exist, and their properties and relations. Since April 2009, the NIC has been building an Innovation bridging ontology, which is known as ‘OHIO’ (Open Health Innovation Ontology). By that we mean that the ontology covers the full life-cycle of an innovation, starting with need, then leading to commissioning, then R&D, and then market entry. This bridging ontology has been crafted through the creation of bespoke innovation ontology components, which have been enhanced with ontologies that are currently widely-used, such as FoF (Friend-of-a-Friend, as used in Linked-In and Facebook) and Dublin Core (used in document classifications).

Triple-Store

In the Semantic Web. an ontology is often expressed computationally as either OWL (Web Ontology Language) or SKOS (Simple Knowledge Organization System). Data, which sits in a triple-store (which is a type of database for the semantic web), is rendered against OWL or SKOS. Following this rendering, the semantically-enhanced data can be disseminated via the tipple-store’s ‘sparkle-endpoint. This semantically-enhanced data can then be expressed via a wide range of applications, many of which are freely available via open-source, such as MIT’s SIMILE and Google. The end result of this semantic pipeline is that the end-user is much more likely to get information that is highly context-specific and meaningful than what is currently the case. The inter-relationships between these components is illustrated below.


http://www.hm-treasury.gov.uk/d/oep_final_report_210409_pu728.pdf

http://www.dius.gov.uk/~/media/publications/O/OLS-Blueprint-Progress

http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/@ps/@sta/@perf/documents/digitalasset/dh_110178.pdf

Data

Note, from the illustration above, that in order for the system to work optimally so that it delivers benefits to end-users, there is a need for high-quality data, ontologies, and applications. Data may come from a wide-range of sources. In order to deliver a quick-win, the NIC has commissioned the curation of semantically enhanced data from:

o Pera (general descriptive data),

o TTP (European networks),

o QinetiQ (future technologies) , and

o Innovation Exchange (technology market assessments.

Each supplier is working to their own unique strengths and are providing data that will contribute to meaningful information via the applications.

It should be noted that the curation of data can be expensive and time-consuming. As part of a global trend, data is becoming increasingly freely available from government as well as private-sector sources. In order to benefit from these developments, this initiative will link to high-value data sources that are already freely available from other trusted sources (e.g. data.gov.uk, PubMed, NHS Network, NHS Information Centre, to name a few). Such data will normally get to the NIC’s triple store via direct feeds. Once in the triple-store the data will be rendered via OWL and SKOS, as described above.

Another way that the NIC will be able to receive data is to have RDFa code (which contains the NIC ontology), which can be imbedded into other sites. The benefit of this is that the data that the NIC’s triple-store receives will have a high degree of specificity. To help propagate RDFa uptake on other sites, the NIC will provide tutorials and free downloads of the RDFa code. In addition to this, all applications that the NIC develops will include RDFa and management tracking code. What this means, practically, is that when an application is downloaded from the NIC site and uploaded onto another site, both the RDFa code and the management tracking code are also downloaded. Note that widgets will be designed to ensure informed consent from end-users – ie the user will explicitly grant or not grant use of RDFa code and management tracking code in the Terms and Conditions of use. The benefit of this will be that the NIC will be able to extend its access to both data and networks, and also produce evaluation metrics to inform the NIC business.


Applications

Recently, there have been big advances in displaying massive amounts of data to make them easily accessible. This is emerging as a vibrant and creative field, melding the skills of computer science, statistics, artistic design and storytelling. Gapminder, for example, enables people to load data into applications, which generate new understanding and innovative ways to communicate.

Example of Gapminder display of data

Having entered data and produced a graph, end-users are also able to share their graph with others. This entire offer is provided via a simple, easy-to-use interface, as illustrated below.

Example of Gapminder Data-Sharing Tool


http://www.economist.com/specialreports/displaystory.cfm?story_id=15557455

One of the consequences of the Gapfinder initiative is that a number of people have started to produce open source libraries that can enact some of these kinds of visualisations. This provides an important opportunity for the NIC to become a Go-To place, where the NIC will:

o build and host compelling semantically-enhanced applications

o provide space for others to create and post their own applications

o enable stakeholders to use the applications to inform their decision-making

o enable stakeholders to share their data outputs with others. This would contribute to

the NIC ontology, increase the amount of data in the NIC’s data hopper, and encourage a community of users.

o provide added-value widgets, which end-users can upload onto their sites. These

widgets will contain RDFa code and management tracking code, which the end-user can choose to enable or not. For the end-user, benefits include having their data (e.g. innovations) contribute to the larger data base and the applications arising from these (e.g. mapping of innovations). For the NIC, benefits include further reach to networks and datasets, and the ability to track and evaluate the impact of the NIC’s offering.

Progress to Date

From April 2009, the NIC assembled a group of leading experts operating in the semantic web space. The NIC’s goal is to exploit the potential of the semantic web to support innovation development in the NHS. The NIC has:

NIC Server on the Cloud

The idea of this was to establish a triple-store in the cloud, running 4Store, and then import the Virtuoso triples into it. Once there, the aim was to allow these triples to be queries using SPARQL. All of this was to be reproducible by running scripts to launch further servers.

The server is running and has been demoed.

There is one remaining deliverable which I'd like to do, which is to move all of the scripts into the new Linked Data Platform project. I'd hoped to do that this week, but it looks like it will have to wait until the end of next week, now.


Application Demonstrators

In order to test the Linked-Data system, the NIC commissioned the development of two demonstrators are able to prove data being obtained from the triple store, via the SPARQL end-point, could be displayed in a widget of some kind.These tests have proved successful. Both of these applications, described in more detail, below, will become part of the widget platform/iStation.

Smaller Demo

In the end we went further and used this demo to (a) illustrate that we could convert the data into other formats, and (b) provide a second visualisation.

The first format that we were able to convert to was the Google Visualization data format, and by supporting that we opened up a whole library of widgets that can be used with the data. The second format that we can convert to is MIT's Exhibit format, which gave us access to a basic faceted browser and a timeline component.

Larger Demo

The larger demo is about linking a number of different data visualisations together.

For this we chose information that could be represented on a map, in a bar chart, and in a stacked chart. As the selection criteria is changed (data range, for example), all of the components on the page change at the same time.

This is more ambitious than the simple demo, because it involves binding components together. But it builds on the first demo, because the basic unit that we want to build the page from is a simple widget.

This means that once the framework is complete we can add any widgets we like to a page, and create quite complex layouts.

This work is pretty much complete. The framework part has been in place for about a week now, but what has held up the final stage is the lack of grouping support in 4Store, which makes it impossible to count things such as 'the number of innovations set to enter the market in 2020, grouped by the setting in which they would be used'.

For now we're working around this by adding that counting mechanism to the JavaScript code, which should be ready shortly.

NLP Crawler

The crawler work is underway. The aim in this project is to install the GATE framework on the same box as the triple-store, and to do it in such a way that it can be split out onto a different server if necessary (NLP processing can be quite resource-hungry, so we need to retain flexibility). Once GATE is installed then some of the GATE plug-ins [1] can be added and configured.

The first step is taking a little longer than anticipated in terms of the configuration scripts, but it should be resolved at the end of next week. Once that is done, the GATE modules I'm proposing to install are:


o Reuters' OpenCalais;

o KEA's SKOS processing. This would be configured to do two things:

o look for MeSH terms;

o look for terms from our own SKOS vocabularies (such as SHAs, scenario locations, and so on).

The reason for picking this collection is as follows:

o OpenCalais provides a good spread of news-related terms, that are by their nature non-specialist, but give a good sweep. This means that we'll pick up a lot of basic information such as countries and cities, high-profile companies and people, universities, and so on;

o KEA can process any SKOS document -- you probably remember that SKOS is a vocabulary language -- and use that to provide terms against which to run the NLP. This is ideal, because when our developer created the OWL ontologies for tmg, he adopted a strategy of using SKOS for any lists of items, such as SHAs, innovation status, types of innovation, and so on. All of these can be fed in to the NLP engine. Also, someone has produced a SKOS version of MeSH, which would be interesting to feed in.

Once the NLP framework is in place, our plan is to 'point' the crawler at the current NIC web-site (competitions, needs, forums, etc.), and one other location, such as New Scientist, PubMed, or something along those lines.

Adding more targets for crawling will be an ongoing task, of course, so ongoing management of the service will need to be in place after the build.

Data Curation

The NIC has commissioned the semantic curation of publically available files. Over 5,500 files have been loaded onto the NIC’s triple-store as of 20 April 2010.

June Deliverable

Linked Data This the new public face of the Linked Data Platform (LDP), and would have the following features:

o built with Drupal (data.gov.uk is also built with Drupal). This makes it very quick to add

modules that provide features such as comments, forums, mailing-lists, ratings, questionnaires, and probably just about anything else one could think of. This would also


make it very easy to manage user registration, access to different areas (even to different widgets), and so on;

o instructions on how to embed RDFa. This work is being carried out separately, but will be

incorporated into this area.

o a widget directory, showing a list of widgets, with ratings, comments, categorisation, etc.;

o a widget embedding page, where users can configure a widget for use on their site, by

choosing a widget type along with a selection of the data that is available in the platform. The result will be a piece of mark-up that the user can then embed into their organisation's web-site, their personal blog, etc. This is a common technique, used in sites such as YouTube;

o the widget delivery mechanism. Once the code for a widget is embedded in an external

web-site, then every time a user accesses the web-site, a request will be made by the end-user's browser to the widget platform, to obtain the widget.

RDFa Tagging of NIC Site

The Needs, Competition, Projects and Showcase sections of the NIC website will be given RDFa tags to enable data generated from these sections to be semantically enhanced and added to the triple-store.

October Deliverable The NIC will focus squarely on building and sharing compelling semantically-enhanced applications to support innovation-related decision-making. The NICs work continues to develop at pace and is advancing very well. Taking a rapid, user-centred design approach, the NIC has kicked-off the next phase of build.

Network Collaboration

In time for the DH Expo event in October, the NIC intends to work collaboratively with national NHS organisations to add-value to their user-communities.

o NHS Networks . Build an application to aggregate need, as expressed by networks of clinicians and managers.

o NHS Information Centre . Build an application to support commissioning of high-impact technologies.


o NHS Scientists . Build a semantically-enhanced social networking platform to support NHS scientists.

o NHS Choices , Deliver semantically-enhanced feeds of NIC Showcase content to this channel. Establish patient-centred needs suggestion form.

o NHS Supply Chain . Semantically-enhance Scorecard tools, enabling data to contribute to the triple-store.

New Needs Section

The NIC will work with the TSB and KTN to deliver the next generation of the NEEDS section on the NIC website. This section will benefit from multiple sources of data, including those listed above. These sources will be combined with the exiting resource, which targets individual clinical statements.

User-Engagement

To reach these goals, the NIC will deploy an Agile development approach in building its ontology and applications build. The aim is this approach is to release both ontologies and applications early and often, and then to engage end-users to identify what is most compelling and what could be done to improve the proposition. The most compelling applications will receive the most attention. Developers will seek to sprint such applications along the development pathway, delivering benefits to end-users as quickly as possible.

In order to reduce risk, save money, and benefit from the earlier work of others, the NIC will leverage word previously produced for the COI http://code.google.com/p/argot-hub/. In this context, note the use of Bundles, Discussion, and Issue-Tracking and Source Control

o Bundles

At the bottom of the opening page you'll see that there is a list of what I called 'bundles'. A bundle is a collection of terms that tend to be used together, but they don't have to come from the same ontology or taxonomy.

For example, the vacancy bundle uses terms from FOAF and Dublin Core, and groups them together to allow the expression of the concept 'vacancy'.

o Discussion At the bottom of this page you'll see that each page can have a discussion about it:

<http://code.google.com/p/argot-hub/wiki/ArgotOrganisation>

This is a very useful way to get input on the ontologies as they are being developed.

o Issue-Tracking and Source Control


http://code.google.com/p/argot-hub/wiki/ArgotOrganisation

http://code.google.com/p/argot-hub/

Note that the argot-hub project has source control (where the ontologies can be stored) and an issue-tracking system (which allows people to flag up problems with the ontologies, if they spot them).

Moving forward, the NIC’s OWL files will move to the argot-hub source control system, and then a wiki page to be created for each area. This will make changes easier to track, and also make it easier to discuss each section of the ontology. The development of the Innovation Ontology is modular and bridging. To explain, the core innovation ontology will be enhanced with ontologies that already exist and in wide use (FoF, Dublin Core, MESH). Where ontologies do not exist and are needed, the NIC will commission new core elements, such as Need, Competitions, and RoI.

April 2011 Deliverable

From mid-Oct to end-Oct, the NIC will conduct a process and impact evaluation of the Linked Data Initiative. The evaluation strategy, which is discussed in some detail, below, is designed with the specific purpose of providing information to improve NIC performance against the NIC’s Goals and Objectives.

3. Assessment of benefitsThe National Innovation Centre (NIC) has assembled a group of leading experts operating in the

semantic web space, with the goal of exploiting the potential of the semantic web to support

innovation development in the NHS.

The aim of the initiative is to find and demonstrate possible solutions to improving how publically

available data can be better used to inform decision-making for key stakeholder groups. To achieve

this, the programme team is building and sharing semantically enhanced applications to support

innovation-related decision-making.

By October 2010, the NIC intends to deliver the following:

o a set of compelling semantically-enhanced innovation tools to support clinicians with

commissioning responsibilities;


o an evaluation of the efficiency, effectiveness and quality impacts of these tools; and

o the establishment of a user-community, which both informs and is informed by these DH

assets.

Aims and Objectives

The main aim is to deliver an evaluation of the efficiency, effectiveness and quality impacts of

semantically-enhanced innovation tools, with the following objectives:

o To undertake an assessment of the NIC Linked Data Initiative Programme, and its component

work streams, to identify:

o success of delivery against programme aims and objectives

level and success of stakeholder engagement with the programme

positive and negative elements of the programme and its component work streams

o To capture learning from the programme and to disseminate this to the NIC team for

discussion

o To provide input into the development of the initiative against intelligence collected, to deliver

increased impact

o To assess and understand the sustainability challenges

o To assess whether the programme has demonstrated an impact

o To assess and understand the cost implications of the programme and identify whether it is

considered to be value for money

Overall Approach

The evaluation will take a formative, before and after process and impact approach, linking closely

with the application development, communication plan and the end user engagement phase.

Broadly, the aims of the process evaluation will be to:

o Identify what is what is working well;

o Identify what is not working well; and

o Inform the desired end state.

The aims of the impact evaluation will be to:

o Identify what the impact been and is likely to be; and


o Identify potentially what the value of the impact is compared with not doing anything

different/investing resources.

The formative approach will build in critical development cycles at key points during the Linked Data

Initiative, in line with the agile application development approach (as shown in Figure 2), which will

enable the programme team to review and reflect on the programme activities, to learn from the

outcomes, and consequently to refocus activities, in order to better implement the initiative. Each

development cycle will build on the other to:

o facilitate the process of feedback to NIC about key findings, learning outcomes and

challenges that have arisen at critical points in the evaluation;

o provide commentary on the process of change throughout the lifecycle of the Linked Data

Initiative;

o provide interim recommendations about how the Linked Data Initiative can increase its

impact;

o provide an assessment of the prospective impacts of the Linked Data Initiative, considering

whether these are likely to be sustained; and

o drive the focus of the research questions, data collection and analysis methods.

Critical development cycles in the agile development of applications


The evaluation will use a logic model approach to provide the framework upon which to develop the

research questions. The logic model will be useful for linking the outcomes with programme

activities/processes and the theoretical assumptions/principles of the programme. It will provide a

systematic and visual way of presenting and sharing understanding of the relationships among the

resources operating the programme, the planned activities, and the anticipated changes or result.

It will also be useful for facilitating thinking, planning, and communicating about objectives and actual

accomplishments, thus informing the NIC’s ISO9001 processes. In line with the critical development

cycle approach, the logic model will be used at key points in the evaluation (e.g. post Expo), to

validate the key assumptions with the programme and project teams about the overall programme

and the component work streams’ inputs, processes, outputs, outcomes.

NIC Linked Data Initiative logic model


Process evaluation

The process evaluation will document and analyse the development and implementation of the NIC

Linked Data Initiative, assessing whether this has been implemented as planned and whether the

expected outputs have actually been produced.

It will consider the quality of the implementation of the programme and its component work streams,

and if it has supported innovation-related decision-making in the short and long term. It will also

assess the logic and implementation of the programme’s objectives, inputs and processes, and

whether the programme has delivered its expected outputs.

The research questions for the process evaluation, which will be refined and agreed with the

programme team, are summarised in the table below with the range of methods we propose

employing in order to address these:

Evaluation

focus

Research Question Methodology

Programme

design /

objectives

Were the programme’s objectives realistic and

achievable?

Were new objectives added and why?

How effectively was the programme designed to meet its

objectives?

Qualitative interviews with

programme delivery team

to review programme

logic model and

programme design (April

2010)

Programme

inputs

Were the right people involved (programme team,

designers, developers)?

What changes occurred in leadership or personnel? What

effect did these changes have?

What costs were incurred? Did they exceed initial

projections?

Survey with programme

delivery team to monitor

the implementation of the

programme (monthly)

Attendance at Heartbeat

meetings (monthly)

Programme

implementation

Have the planned businesses processes been

implemented?

Do all elements of the project work coherently?

What problems were encountered in implementing

objectives? How were they resolved?

Has delivery been in line with the programme plan?

Have all planned activities been implemented? If not,

what remains to be done? Were they accomplished on

schedule?

If objectives, plans, or timetables were revised, why was

this necessary?

Survey with programme

delivery team to monitor

the implementation of the

programme (monthly)

Review of programme /

project plans in line with

agreed route map

(monthly)

Attendance at Heartbeat


Were the methodologies appropriate for developing

possible solutions to improving how publically available

data can be better used to inform decision-making for key

stakeholder groups?

How effectively have the project streams contributed to

the delivery of the programme?

meetings to identify what

is / is not working well and

what needs to be

changed (monthly)

Programme

management

Has the programme management stream been effective?

Has the NIC assured delivery of the programme?

Has the communication work stream been effective in

engaging and informing user communities?


the programme, quality

assurance, and

communications leads

(monthly)

Programme

approach

How effective has the rapid, user-centred design

approach been?

How effective has the establishment of the user-

community been?

How effective has the 3 stage Agile approach been?

Did the initiative identify users’ needs?

Did the initiative effectively develop solutions to meeting

need?

Has the initiative been successful in sustaining user

value/interest?

Matrix input into user

testing (monthly)


clinicians (August /

September)

Delivery of

programme

outputs

Have outputs been delivered within agreed timescales

(i.e. by Expo in October 2010)?

Has the programme delivered the following outputs:

- To have started development of 9 application

(minimum) at beta level and the establish of a user

community for each via Open Labs

- Release of 3 applications past beta, with an

average of at least 100 regular users per day from

an influential market

- Three influential champions who are motivated to

act on behalf of the initiative

- To have a business process which is explicit and

reproducible

- A business case for further investment (including

ROI calculation in terms of numbers of adoptions of

new technologies which are likely to happen earlier

that would not have happened otherwise)

Is the quality of outputs of a high standard?

Review of programme /

project plans in line with

agreed route map

(September / October)

Assessment of the extent

to which the programme

outputs, as specified,

have been delivered

(September / October)

Impact Evaluation

The aims of the impact evaluation will be to identify what the impacts of the programme have been


and are likely to be and to identify potentially what the value of the impacts are compared with not

doing anything different/investing resources. It will assess whether the programme has been

successful in meeting its stated aim of creating the “Go To” trusted source of technology innovation

information which is routinely used in decision making in the NHS”.

The first step in the impact evaluation is mapping qualitatively what the outcomes and likely impacts

might be. This will be achieved through discussions with the programme team, in line with the

development of the programme’s business case.

The research questions for the impact evaluation, which will be refined and agreed with the

programme team, are summarised in the table below with the range of methods we propose

employing in order to address these:

Evaluation

focus

Research question Methodology

Outcomes:

NIC

What are the benefits that are being delivered Vs a do

nothing or other options (i.e. what value is the investment

creating?)

Has the NIC been successful in developing tools that will

find and link relevant data to give more meaningful

results?

Has the NIC exploited the potential of the semantic web

to support innovation development in the NHS?

Have the semantically-enhanced applications delivered

value in the health innovation space?

Has the initiative accelerated adoption of vfm and low risk

technologies?

Has the take up/usage been what was proposed?


programme delivery team

to review programme

logic model

(September /October)

Programme team

assessment of

programme outcomes

(Final Heartbeat meeting)

Website hits using Google

Analytics (ongoing)

Outcomes:

user

community

Did the initiative meet users’ needs?

Has the initiative helped innovators and clinicians find the

answers to all their questions about innovation related to

people, places and technologies in their respective fields?

Has it helped clinicians discover more meaningful

information about Healthcare technology innovation when

using the web?

Has the programme assisted with behaviour change – are

clinicians doing anything differently?

Has this initiative been successful in supporting

innovation-related decision-making? Has it increased

rationality in decision making processes?

Has the initiative been successful in supporting clinicians

with commissioning responsibilities?


clinicians (August /

September)

Impact What are the likely impacts of the programme?

Will the initiative improve performance in healthcare?

Is the programme likely to have the following impacts?

Programme team

assessment of potential

programme impacts (Final


- Improved outcomes

- More delivery of care at home

- Improved patient experience

- Improved staff satisfaction

- Reduced hospital utilisation

- Improved use of resources

Heartbeat meeting)

MethodologyThe evaluation will employ a variety of methodologies, both quantitative and qualitative, as outlined

below:

Interviews with the programme team

The qualitative interviews with members of the internal delivery team will form an integral component

of both formative and summative elements of the evaluation. The interviews will be used to monitor

objectively how well the programme is being implemented and will therefore be conducted by a

member of the Matrix team who has not been involved in the initiative to date.

We will conduct a set of in-depth qualitative interviews with all members of the delivery team in April,

to review the programme’s logic model and the design of the programme. These will be either be

face-to-face or telephone, whichever is most convenient for participants.

We will then conduct a set of in-depth qualitative interviews in September / October to assess the

outcomes of the programme, the benefits it has delivered, the success of the tools that have been

developed, and how effective the NIC has been in exploiting the potential of the semantic web to

support innovation development in the NHS.

Additionally we will conduct monthly telephone interviews with the programme, quality assurance and

communications lead to monitor whether the programme management stream been effective, whether

the NIC has assured delivery of the programme, and whether the communication work stream been

effective in engaging and informing user communities.

Survey with programme team

In addition to the qualitative interviews we propose administering an online survey to the programme

team on a monthly basis to monitor the implementation of the programme.

The findings from the survey will be analysed for presentation at the monthly Heartbeat meetings, to

assess the extent to which implementation has occurred as planned, problems that have been


encountered in implementing objectives, whether delivery is in line with the programme plan, and how

effective each component work stream has been in contributing to the delivery of the entire

programme.

Interviews with the user community

The implementation team will have contact with the clinicians on a weekly basis. Therefore to reduce

potential burden on the user community, we would propose suggesting a set of additional questions

for the delivery team to ask to monitor the effectiveness of the agile approach and the stakeholder

engagement stream. The findings from these interviews will feed into the formative element of the

project to address how effective the user-centred design approach has been, whether the initiative

has identified their needs and developed solutions to meet their needs, and whether the initiative has

been successful in sustaining their interest and value.

As part of the outcome evaluation, in August / September we would propose conducting a set of

interviews with a sample of the user community, either by telephone or face-to-face, to assess the

extent to which the initiative has met their needs, whether the programme has assisted with behaviour

change, and whether it has assisted with the decision-making process.

Website hits

We will use Google Analytics to track visitors to the websites. Google Analytics is the enterprise-class

web analytics solution that gives rich insights into website traffic and marketing effectiveness:

http://www.google.com/analytics/. This will enable us to find out how many people visit, how long they

stay on the website, which pages they visit, where they came from (Google search, a link from

another site etc), which country they are based in, and numbers of new and returning visitors.

Timeline


http://www.google.com/analytics/

Cost / Benefits Assessment

This will clearly identify the cost benefit against the investment that is being made. This should consist of a simplified presentation of the financial cost/benefit analysis - presented in tables or graphs;

This should include an assessment of:o Investment value – Revenue and Capitalo Cost Saving first 12 months (£)o Cost Saving after first year (£)o Depreciation costso Total cost (£)o Cash outlay (£)

Additionally, an assessment of unquantified Benefits and Costs, and Strategic Contribution should be considered:

o Try very hard to quantify all costs and benefitso All direct costs should be includedo Indirect costs should be includedo If a non-financial benefit is significant, then define ito

Make the impact of the benefit tangible – describe all likely effects and implication

Dissemination and reportingA monthly evaluation report will be generated by the Programme Team, and published on PMPT.

These reports will use a standard format and will present the key findings, learning outcomes and

challenges that have arisen at critical points in the programme, provide commentary on the process of

change throughout the lifecycle of the programme, and provide interim recommendations about how

the Linked Data Initiative can increase its impact. These will be presented using Power Point and will

be uploaded onto the Linked Data website for consideration by the programme team.

The final phase of the initiative will be completed on completion of the Expo in October. At this time, a

report will be produced to document process and impact evaluations. This will identify what the impact

been and is likely to be; and what the value of the impact is compared with not doing anything

different/investing resources.



Education

NIC Linked Data: the OHIO project