Trials Units Information Systems General Proposals … · Trials Units Information Systems – General Proposals ... Introduction ... and dissemination of software development projects,

Trials Units Information Systems –

General Proposals

Data and Information Systems Project

The DIMS Project Team

Trials Unit IS – General proposals

The DIMS Project Team Steve Canham Database / Systems Manager, Institute of Cancer Research Clinical Trials

and Statistics Unit

Jim Charvill Technical Design Architect, National Institute for Health Research

Will Crocombe Information Systems Director, Clinical Trials Research Unit, University of Leeds

Richard Farr Project Manager, National Institute for Health Research

Acknowledgements Steve Canham would like to express his gratitude to colleagues at the Institute of Cancer Research’s Clinical Trials and Statistics Unit, and in particular to the Unit’s Director, Professor Judith Bliss, and … Will Crocombe would like to express his gratitude to colleagues at the Clinical Trials Research Unit, University of Leeds, and in particular to the Unit’s Director, Professor Julia Brown, … for their support and forbearance over recent months during the absences occasioned by their work as part of the DIMS project team.

Trials Unit IS – General Proposals

Contents

Introduction ................................................................................................................................ 1

Current Issues ............................................................................................................................. 3

Current Strengths........................................................................................................................ 4

Previous Work and Lessons Learnt ............................................................................................. 6

A Strategy for the Future ............................................................................................................ 8

A Standards based approach............................................................................................................... 8

Implementing System Standards....................................................................................................... 10

Implementing Data Standards........................................................................................................... 11

Implementing Modularity.................................................................................................................. 13

Implementing Support Structures ..................................................................................................... 15

Governance ....................................................................................................................................... 16

Costs and Benefits..................................................................................................................... 16

Conclusion................................................................................................................................. 22


Page | 1

Introduction

1 In 2008 the National Institute for Health Research and UKCRC instigated the Data and Information Management Systems (DIMS) project to investigate data management systems in Clinical Trials Units (CTUs) and to propose a strategy for the future development of such systems.

2 The project is one of the ways in which the NIHR is working with the UK Clinical Research Networks to realise the vision set out by the government in Best Research for Best Health1, and forms part of Implementation Plan 1.2 (the Information Systems Programme) of the Best Research for Best Health strategy.

3 The DIMS project is timely because there is currently a great deal of uncertainty in trials units about information systems, for instance about the introduction of remote data capture systems or using data standards. Operational requirements and standards are poorly defined and there has never been a co‐ordinated approach to system procurement. Consequently, each trials unit has adopted their own technical and organisational approach to IS, and we now have a highly heterogeneous mix of systems, data structures and technologies, impeding the efficient sharing of both data and expertise. That in turn has led to wasteful duplication of effort even while the costs of the systems (licences, maintenance etc.) and the demands upon them (e.g. because of an enhanced regulatory regime) appear to be rising sharply.

4 A clear set of standards and requirements for information management systems and the data held within them, and a road map for the future development of such systems, is therefore a high priority issue for CTUs and their funders alike. The need was underlined by feedback from the UKCRC CTU Registration Committee, which highlighted the fact that, with different units in very different stages of organisational development, the quality of information systems available to trialists varies considerably.

5 Recent projects in this area have not succeeded in clarifying issues, perhaps because they have been too limited in scope. The DIMS project, on the other hand, offers an opportunity to examine the entire information systems landscape in clinical trials. At issue is not just how such systems can be best designed, procured, assembled, and funded. More fundamentally, DIMS provides the opportunity to consider how information systems can best support the research activity of the units.

6 The specific aims of the Data and Information Management Systems (DIMS) project are to:

a) review current data systems and information standards with clinical trials units, b) inform the development of a nationally agreed set of data and system standards, c) scope and detail functional IT requirements within clinical trials units, and

1. 1 Best Research For Best Health: A new national health research strategy, Department

of Health (Research and Development Directorate), Jan 2006 2.


Page | 2

d) establish options for a strategy and standard policy for data and information management.

7 The first three aims have been addressed by the production of four white papers. These are:

a) Trials Units IS – Results of Consultation…which collates and discusses the responses received from a questionnaire sent to all UKCRN registered units, in late 2008, asking for details of their current systems and their concerns for the future.

b) Trials Units IS – Data Standards…which describes the rationale for using data standards

and makes detailed recommendations as to which standards could and should be used in the future and how they might be introduced.

c) Trials Units IS – System Standards…which specifies standards of good practice and procedures for both functional and infrastructure components of IT systems in trials units, as a guide to assessing compliance with regulatory requirements.

d) Trials Units IS – Functions and Modules…which identifies and describes distinct

functions for IT within trials units, describing the inputs and outputs associated with each, and which also presents the case for a much more modular approach to IT systems in trials units in the future.

8 This paper is designed to meet the final objective, i.e. to propose a general strategy that

represents the best way of improving efficiency, quality and consistency for trials units IT, and that can be supported both by national agencies and the trials units themselves. It is based on and draws upon the more detailed facts and arguments set out in the four white papers listed above. In essence the strategy is to:

a) Agree and maintain a set of functional requirements and operational standards for trials unit information systems, so that units can assess their own systems, developing them where necessary, and so that funders and co‐ordinating organisations can be assured that data is being managed in an appropriate fashion.

b) Adopt or co‐develop various data standards to better support data sharing and inter‐operability, both within and between units, to improve the speed with which systems are set up, to more easily capture data from a variety of sources (including, ultimately, from electronic health records) and to support more flexible procurement.

c) By using standards, to specify trials information systems as an assemblage of inter‐

operable modules, and to work with both system vendors and developers in the trials unit community to allow future systems to be built or bought (or both) that way, in a modular fashion, according to local requirements, resources and preference.

d) Develop and maintain the tools and organisational structures needed to support the

introduction of operational and data standards into units, including where necessary


Page | 3

the specification, co‐ordination and dissemination of software development projects, as well as procurement from commercial suppliers.

e) To maximise impact and leverage and to reduce costs, to work as collaboratively as possible in all the above – with the pharmaceutical industry and with each of the home nations as well as with NHS initiatives – and for the governance and funding structures of ongoing work to reflect that collaboration.

Current Issues

9 The central issues facing trials units with respect to their information systems have already been mentioned – a generally unproductive heterogeneity, a lack of standards, a lack of central leadership in the past, rising demands and rising costs. To better characterise the current situation, however, the DIMS board requested a consultation exercise involving all 40 of the trials units currently registered with UKCRC.

10 A questionnaire, covering a wide range of issues and comprising over 50 questions, was duly sent out in late 2008. 28 responses (70%) were obtained by the turn of the year. A much more detailed description of the results is given in the Trials Units IS – Results of Consultation white paper which also discusses some of the implications of those results. The more important issues that emerged, however, are given below

a) Over 70% of responding units used clinical database systems developed in house, for part or all of their trials, and penetration by the commercial sector was largely but not exclusively centred on one product (InferMed’s Macro). Several units use both home grown and commercial systems, and several units also used legacy systems for older trials.

b) Administration systems are often piecemeal and almost always developed in house.

This includes relatively critical systems, e.g. to support pharmacovigilance, as well as more ‘routine’ trial and data administration, contacts management etc. Links between administration systems, either inside or between trials units, or between the units and the centre or units and trusts, are rare despite the fact that they all potentially make use of the same data.

c) Randomisations systems, reporting and data extraction systems are almost always

developed in house and in isolation, despite the obvious similarities in need between different units.

d) The average ratio of IT staff to total trials unit staff is about 10%. It is estimated that

there are about 80 individuals working as programmers in trials units across the country, with a further 120 working on database construction, scripting, testing, network and user support tasks, etc. Total costs of IT in UK academic trials units are difficult to estimate exactly, but probably include salary costs of £8.4 million p.a., of


Page | 4

which 40% is used to support system development work and programming. (Details and derivation can be found in the Consultation Results white paper).

e) Despite the large pool of labour theoretically available across the country, the

historical pattern of isolated development has combined with recent increases in expectations (e.g. regarding validation and audit), exacerbated sometimes by difficulties in recruiting and retaining staff, to produce considerable pressure on IT staff in many units.

f) Costing and funding methods for IT input are variable but largely based on the

intrinsically inaccurate ‘per project’ model. There is no generally accepted model for estimating costs of IT input.

g) There is considerable interest in eRDC, but involvement in it for most units tends to be

either (almost) all or (almost) none. Some units report great success with eRDC, others feedback frustration. The costs of user support, exacerbated by having lots of different systems, is one of the factors behind the difficulties experienced; others include the costs of licensing and the lack of a clear national strategy in this area.

h) Quality assurance measures and system standards present a mixed picture. Some units

have well developed systems, others are still developing them. The levels of resource required for testing and validation, full change management etc. are poorly defined, whilst the levels actually implemented vary enormously.

i) There is a clear feeling that more information and guidance is required, from the

centre, particularly with regard to compliance and standards – a request in other words for definitive benchmarks to work against. There is also a feeling that there should be better leadership: a clear strategy and policies at national level, both generally and in terms of specific issues such as eRDC, data standards, purchasing etc.

Current Strengths

11 Amidst a relatively long list of problems and issues it would be easy to forget the considerable strengths that also exist amongst the IT staff of the academic trials units, or which derive from the context in which they work. On a more positive note therefore, some of those strengths are listed below:

a) A strong and active IT staff community exists, one which has gradually developed since 2003, and which has now evolved into the UKCRN IS working group (ISWG). Not all units are currently involved in this group, though the introduction of more focused sub groups may improve that situation, but the ISWG provides an effective mechanism to share experiences and ideas, and must play a key role in implementing any strategy for the future.


Page | 5

b) The considerable experience that now exists in modern database management systems for clinical trials, whether commercial or built in‐house, and the ways in which they need to operate to conform to GCP and the general regulatory framework. What is lacking is the organisation of that experience into a coherent set of national standards, but the knowledge is there, and covers a very large range of methodologies and disease types.

c) There is a considerable degree of technical expertise and innovation amidst the

academic trials units. For instance various eRDC solutions, employing different technologies, have been built in‐house and are in use, and there are a variety of sophisticated randomisation systems in place. It will be important for any future strategy to make best use of this expertise, co‐ordinating and securing it as appropriate.

d) The advent of the NIHR and its role as a central, co‐ordinating body for health service

research. This has significantly changed the organisational landscape and provides, for the first time, a mechanism for resourcing and providing central leadership and guidance about trial IS practice. In the past initiatives have been relatively isolated and confined to specific topic networks or geographical regions – in contrast the NIHR provides the opportunity to take the whole picture, develop a comprehensive strategy for trials IS, ensure it is integrated with other related developments, and then underpin it with the resources of a national initiative.

e) The general development of research support systems in the NHS and elsewhere (e.g.

the Research Capability Programme (RCP), the use of SNOMED CT for clinical coding, the common user interface for NHS systems) all offer exciting opportunities for clinical trials systems as well as underlining the need to review how those systems might function in the future.

f) Other countries and organisations are faced with very similar issues as those described

for UK academic trials units. Consequently there is other work – in Europe (e.g. ECRIN), in the US (e.g. CaBIG) and globally by pharmaceutical companies and others (e.g. CDISC, HL7, BRIDG) that are potential sources of systems, standards and collaboration.

g) The DIMS project itself, as the first serious attempt to derive a national strategy for

trials information systems and as a model for future collaboration between the various stake holders, is a powerful positive development. The DIMS board could easily develop into a standing oversight committee for clinical trials information systems.

12 In short, though the problems and issues to be faced should not be underestimated, there exists considerable ability and experience within the trials community, and potentially substantial support and commonality of purpose outside it.


Page | 6

Previous Work and Lessons Learnt

13 Three previous initiatives, only one of which has so far achieved any degree of success, illustrate some of the issues that can influence and even derail general attempts to improve trials information systems. They do provide some useful lessons, however, as well as background and context for the current proposals.

14 The NCRN Initiative: Starting in 2002 and running through to 2006 the National Cancer Research Network explored the feasibility of its (six, later nine) accredited units all using the same clinical database management system. After extensive specification of requirements and testing the product that was selected was Medidata Rave, a web based eRDC solution, used extensively in the pharmaceutical industry but at that time by no‐one amongst academic trials units.

15 In fact the proposal was not just to use the same system for data storage, it was to centralise the storage as well, using a server farm in a ‘bunker’, run by BT, attached at one end to the N3 backbone and at the other, via the joint academic network JANET, to the academic trials units.

16 The next stage was to ask for volunteers to run a pilot, but this proved difficult. No additional funds were forthcoming, at least initially, to support what would have been a duplicate set up and data entry processes, as no‐one wanted to entrust a trial entirely to a completely new system. Many units were wary of eRDC and what it might mean in terms of supporting large numbers of staff, and many were extremely suspicious of the ‘central bunker’ idea and what that might mean for access to and custodianship of ‘their’ data.

17 Most critically, there was little real involvement of senior trials unit staff, who therefore tended to see the initiative as a largely technical exercise, emanating from people they had never heard of, and one that they had neither requested, nor could control, nor, ultimately, become excited about.

18 Only a few units came forward to participate in the pilot and the set up process, even when involving additional support from the vendor and BT, took several months. When time was finally called on the project only a handful of patients had been entered onto just a couple of systems, and there was insufficient data to carry out an evaluation. A great deal of money had already been spent by that point.

19 The Scottish Periscope Project: Discussions related to this project are still ongoing so the final outcome remains uncertain. Periscope (from ‘Project to create an e‐Health Research Infrastructure for Scotland: Phase 1’) arose from a proposal submitted in April 2006 to the Scottish Chief Scientist's Office.

20 The proposal had been developed by a working party that involved representatives of the trials units, clinical research facilities and NHS R&D offices in Scotland. The project proposed among other things to: ‘Develop a clinical studies toolkit to support conduct of clinical


Page | 7

trials and epidemiological studies’. The Scottish Government decided that the development of the toolkit should be put out to open tender. To date no meaningful progress has been made and the procurement process has come to an end without resolution.

21 Cancer Grid: The CancerGrid project began in 2005, with funding from MRC. Five universities were involved ‐ Cambridge, Oxford, UCL, Belfast, and Birmingham ‐ with Oxford leading the software engineering activity. Cambridge (CR UK Cancer Research Institute) and Oxford (Software Engineering Programme) have continued the work with further funding from CRUK and Microsoft Research.

22 The aim of the project was to develop open standards for cancer informatics. These were originally to be promoted using ‘grid’ technologies, but these technologies proved both immature and impractical. The toolkits were relatively unsophisticated, requiring considerable programming effort to duplicate the functionality of applications that were already available to the target users.

23 The development activity was thus focussed upon the production of software that worked to extend and configure applications that are widely used and available within the NHS (and, indeed, throughout government and industry): specifically, MS Office and SharePoint Server.

24 Along with a meta‐model (or model template) for clinical trials, based upon the CONSORT statement, and work on the classification and registration of clinical trials, the project has produced an ISO11179‐compliant metadata registry, a ‘semantically‐aware’ trials design /management system, and a toolkit for clinical data transformation and integration.

25 These models and software applications have been tested through initial deployment on a small number of clinical studies in Oxford and Cambridge. The metadata registry software has been adopted for use within the US caBIG initiative, and is being evaluated by a number of organisations within the UK. Development continues at Oxford, with clinical collaboration in Cambridge and London, and technical collaboration with the caBIG team in the US.

26 The project's focus upon the requirements of clinical researchers, and the recognition that these requirements can be met partly through the (automatic) enhancement and configuration of office applications, has led to changes in attitudes. As one team member put it: “We used to be hung up on open source; now we're really focussed on open standards”.

27 Taken together, these previous attempts to rationalise trials unit information systems would seem to suggest that:

a) Trying to simply change, wholesale, the systems people use is inherently fraught with difficulty – units are faced with giving up what they know (and may have invested heavily in, especially if they have developed it themselves) or running two or more systems side by side. Given the huge amounts of existing and legacy data that the


Page | 8

units would have to manage any system change would take many years, or be very expensive, or both.

b) Trials units have to feel involved in and knowledgeable about any proposed change,

and they should feel they have as local control as is reasonable: no‐one likes being railroaded. To try and force change would generate so much friction – just wasted heat and energy – that any project would grind to a halt.

c) Software tendering exercises in this area, for either existing commercial packages or

bespoke systems, are not an easy or particularly effective option. Given the money spent both on the NCRN scheme and on Periscope in Scotland it is fair to say that, in many quarters, there is no longer an appetite for a large scale tendering option, on either side of the border.

d) Issues should not be conflated together. Introducing centralised hosting is a quite

different issue from a common database system, and people should have the opportunity to reject one and accept the other, not take or leave both as a package. In other words the different components of any solution should not be tightly coupled together.

e) Change needs to be resourced, either directly with extra funds for extra work or

indirectly by the provision of tools and support. Otherwise it will simply not happen – there is no slack in the system. At worst, in the long term, any proposed changes should be cost neutral. At best they should save some money.

f) Technology in and of itself is never going to produce a solution, especially if the techniques used are still at the ‘bleeding edge’. In fact, even relatively cheap and common products like Microsoft Office and SharePoint, if their use is supported by appropriate data repositories, standards and automation, can deliver large portions of the functionality required.

A Strategy for the Future A Standards based approach

28 Taking on board the lessons learnt from past initiatives, together with the current needs of the units, all the indicators are pointing in the same direction: towards standards. To the seemingly endless debates about which system should be built or bought (and there are strong proponents of each approach) the only reasonable response would seem to be any system you like, as long as it can do specified things to specified standards.

29 Fundamentally a trials unit information system is about managing the data to support the science. It is the content that is crucial, not the technology. How that content is stored and structured, exported and shared, validated and secured, described and discovered, are the


Page | 9

key issues. Again the answer is standards, for data rather than functionality, but not the proprietary standards defined by a particular software vendor. Instead they have to be the internationally recognised standards that best support inter‐operability with source systems and most easily allow data sharing.

30 Standards have intrinsic value in their own right, as the next two sections and the two relevant white papers make very clear. But they also emphasise that some previous initiatives have tried to answer the wrong question. It is not ‘which system or approach is the best one to use?’ It is instead ‘what do we want all systems to be able to do?, possibly followed by ‘and so how do I upgrade my own system to make sure it does that?’

31 Critically, a standards based approach allows local choice and control over systems and deployment. If outsourcing the entire IT function works for a trials unit then they would be free to do that, as long as they were assured that their contractor was meeting the required standards. If people want to buy all their functionality from one supplier that is absolutely fine, as long as it meets the functional requirements. If people want to upgrade their existing systems by building and bolting on new modules, that too will work, as long as all the standards are met.

32 This enhanced flexibility of procurement that a standards based approach provides, and the greater competition that should result, is one of the other great advantages of the approach, (something that is discussed in more detail in the Functions and Modules white paper).

33 A standards based approach is not without cost – standards do not write themselves and the process of introducing them will require much debate and explanation, support and training and – as described below and in the Data Standards white paper – tools and the resources and systems to maintain those tools, as well as continued involvement in standard development.

34 The contention is, however, that a standards based approach would at least avoid the massive upfront costs commonly associated with large software procurement projects, and it would save units from being too tied to the foibles and fortunes of individual system vendors. Individual units would instead have much more flexibility to implement enhanced functionality on a piecemeal basis, in line with their resources and local needs.

35 In addition, because standards allow the sharing of both development work and the modules that result from that work, there is much more scope for the academic trials community as a whole to generate and share components (assuming they are properly secured, documented and validated) and the real possibility that the current levels of duplicated effort can be drastically reduced.

36 We are spending something like £3.4 million a year on development activity now. With the anticipated expansion in clinical trials that figure could double in 5 years time if we do nothing to improve the co‐ordination of that activity. Fortunately, using standards to rationalise development goals, co‐ordinate activity and, ultimately, to enable shared components, there is no reason why pro rata development costs could not be reduced by


Page | 10

20%, 30% or even 40% in 5 years time. The per trial IS costs in other areas of activity could also be reduced. (Potential costs and benefits are considered in much more detail later in this document).

37 A proportion of the money saved might have to go into more consistent implementation of quality standards and (for instance) in improving the validation regimes in trials IS. Overall, however, standards should allow us to both grow capacity and improve quality in the most efficient manner.

38 If units would prefer to buy rather than develop their components, ideally selecting from a range of vendors (because the products of each would all have to be meeting the same functional and data requirements) then that would remain an option. If they are prepared to sell products on a modular basis rather than as whole systems, more vendors could enter the market and competition would help to keep prices at a reasonable level. The point is that with standards there is a choice. Without them we remain stuck where we are.

39 The main components of a standards based approach to information systems would be the introduction of system or operational standards, the development and introduction of data standards, the elaboration of a modular approach to system procurement, the creation of an organisational structure and resources to support standard development and use, and finally the setting up of appropriate funding and governance structures. The process of implementing each of these are described in more detail in the sections that follow.

Implementing System Standards

40 Introducing system standards ought to be relatively straightforward and relatively cheap. The systems standard white paper describes how standards could be categorised and reviewed and lists a preliminary set for discussion. It is suggested that an initial self assessment exercise is done to test the validity and practicality of the initial list, not least because this would provide units with an initial opportunity to influence the standards that were set.

41 Because the main function of the system standards is for the units themselves to identify and introduce good practice, it should be the units themselves that review and maintain those standards. It is suggested that a standards review sub group, put together under the auspices of the current IS and QA UKCRN working groups, could be set up to carry out this function.

42 Funders, auditing and co‐ordinating organisations (e.g. MHRA, NIHR) also need to be assured that data is being managed in an appropriate fashion, and so would need oversight of the standards and the ability to request changes, as well as receiving the results of assessment. The funders’ attitude towards, and use of, these standards will be crucial in determining how effective they can be in improving the quality of information management in trials units.


Page | 11

43 How the standards would be linked to accreditation / inspection would be a matter for the relevant auditing organisations, but initial meetings with MHRA should take place so that units know how the standards would compare to (or even be used as the basis for) the expectations on IS during an inspection.

44 It is envisaged that system standards would need regular review – say six monthly – for instance to accommodate changing technologies and regulations, or, with sufficient notice, to ‘notch up’ the expected quality level in a particular aspect of trials systems, if that was felt to be reasonable and generally attainable. Regular review also allows ‐ again with proper notice and support – data standards and other components not widely supported at the moment to be added to the requirements list, thereby promoting other parts of the overall strategy.

45 The most contentious issue would be what happens if a unit does not meet the standards associated with a high quality information system, and cannot easily close the gap. One option would be to seek additional resources and improve the local systems. The second would be to outsource part or all of the IS function.

46 Information Systems are, ultimately, a service to the core scientific effort, and are not an essential part of a trials unit, even though historically most units have ended up growing their own IS teams. A unit might, particularly if it was relatively new and / or small, examine the standards that were being asked of it in terms of IS and simply decide it wasn’t worth it; that it would be more efficient to ask a larger unit down the road to carry out the IS function, for a fee, including having the data hosted remotely.

47 Several units already operate in this way, offering services to small academic research teams or in a few cases to the pharmaceutical industry, effectively operating as the IS component of a CRO. There would need to be a battery of agreements in place to cover (for instance) confidentiality, security and access, but at the heart of those would, once again, have to be standards.

48 In a few cases, a funder that was concerned about the perceived quality of IS in a very small unit might even insist on such an arrangement, as being the simplest and cheapest way of ensuring an adequate quality of information service for a research project. Using system standards might, therefore, alter the distribution of information systems in clinical trials, though there is no question that the research effort itself would have to move.

49 The introduction of a recognised list of system standards is fundamental to what the DIMS project is trying to achieve, and given the expressed wish of many trials units for just such a list and related guidance, the scheme outlined in this section, and described more fully in the relevant DIMS white paper, should be implemented as soon as possible.

Implementing Data Standards

50 Data standards are a more challenging issue than system standards, not least because there are many different types of such standards, relevant to different part of a trials unit’s IS. The proposed standards are divided into four groups, those dealing with


Page | 12

a) Clinical Databases: The subject data that is sought, and when, the ways the resultant questions are defined and coded, the way data collection is structured and the way responses are coded and analysed.

b) Trial Description: The trial’s overall aims and type, its identifiers, sponsors, sub‐

protocols etc; its schedule of visits, milestones and data collection events; and the detailed descriptions of each data item.

c) File Management: The ways in which data is packaged, formatted and transmitted

between different stages of the trial process.

d) Trial Management: Administrative data such as people and organisations and their contact details, milestones in the trial approval and funding processes, and the research governance and agreement data of centres, safety reporting and the resulting data flows between trials units, investigators, and regulatory authorities

51 The white paper on data standards describes in detail the standards that ought to be

introduced in each of these areas and why, and includes a long list of specific recommendations as well as a summary of the immediate actions that ought to take place in the first 12 months of any programme, either because they provide a ‘quick win’ or because they prepare the way for later more substantial development.

52 Most data standards are still in development and there is wide variation in the maturity of the different systems. Involvement in the various Standards Development Organisation (SDOs), or their agencies in the UK, will therefore be an important ongoing part of introducing these standards, to ensure that the needs of trials units are represented as the systems evolve.

53 SDO involvement is but one example of a range of collaborative activities that will be needed. Others include participating within European groups (e.g. ECRIN) and – in particular – working with pharmaceutical companies. The latter could be very important in the context of developing a ‘metadata repository’ of clinical data items, for this is something that some pharma companies are also embarking upon, at the same time as we have recognised the need for such a repository within academic units.

54 It is estimated that the standards themselves – most of which would be in the form of XML schemas – could be established in three years. Linking data to them may well take longer, for instance the gradual generation of structured descriptions of existing trials, or even continue indefinitely, as it would with the gradual accumulation of items in the metadata repository mentioned above.

55 Introducing data standards will involve the units in some additional work. There will be a need to reduce this work to a minimum, and that will take central support and relevant tools, guidance and training as well as, in many cases, specific input to identify the best way of implementing standards in a particular unit. The development of tools and related central stores, for instance to allow easy selection of standard data items, or easy update of


Page | 13

standardised contact systems, is described in more detail in the data standards white paper.

56 Even though much of this work could be combined with other existing projects, such input represents the main costs of introducing data standards, because people would need to be employed specifically to write and provide the guidance, or (for instance) to help manage a metadata repository. There would also be the more general involvement in standard development work, and a general managing / co‐ordinating function is also required to drive the data standards initiative onwards. Though only a handful of people would be needed, these costs underline the need for collaboration with as many different organisations, and therefore potential funders, as possible.

57 Despite their cost, data standards do bring enormous benefit in their own right. Introducing them should…

a) Increase the scientific and medical value of clinical trial data by making the data generated by trials easier (and quicker and cheaper) to locate, characterise, share, and compare.

b) Make trials quicker and easier to set up by using off the shelf question sets from a

common library, common controlled vocabularies for coding, and encouraging common data structures and design philosophies.

c) Make trial administration more consistent and efficient, and allow the specification

and construction of tools designed to support these processes more transparent and efficient.

d) Allow data linking with non trial sources, in particular (eventually and probably

indirectly) routine clinical data. Using electronic health records will be dependent, however, upon systems in trials units using the same codes, the same XML schemas etc., as those in the source systems.

58 Even on their own terms therefore, data standards represent a worthwhile investment. Add to

that the potential they have for modularising systems, increasing competition and reducing duplication of development and they can more than pay for themselves. They have always been fundamental to the DIMS project, as they are to the wider Better Research for Better Health programme, and the recommendations listed in the data standards white paper should be implemented.

Implementing Modularity

59 The advantages of a standards based modular approach to specifying, developing and purchasing systems have already been stressed. Such modularity will only be possible, however, if we, the academic trials community, develop the necessary components ourselves or convince system vendors that if they want to sell their products to the NHS and UK academic units they need to present them as standards compliant modules.


Page | 14

60 It will therefore be important to involve system vendors in discussions as early in the process of adopting / developing standards as possible, not least because some vendors have already started to support some coding systems (chiefly MedDRA) and the evolving CDISC standards. Collaboration with pharma would also be important here, if possible, because that would increase the collective bargaining power with vendors.

61 If all the relevant standards and schemas can be finalised within three years, as proposed, and allowing for a further two years for relevant development, then five years on from the beginning of the process, in many instances less, units should have the ability to purchase / upgrade systems in a modular fashion.

62 Similarly, in a parallel development exercise, local developers in trials units may also have produced modules that meet the standards specified. Such modules, produced in isolation by individuals or very small teams, would not normally be considered for wider use unless

a) The source code and all documentation had been signed over to a central agency (e.g. NIHR), and that remained the case for all subsequent versions. Formulae will therefore need to be developed to deal with IP rights, though it is hoped that, at least within the academic community, sharing between units can remain low or zero cost.

b) The system had been validated, at least technically, and that remained the case for all

subsequent versions. Again, there will need to be clarity about the potential impacts of any local modifications on validation. For that reason a traditional open source model might not be the most appropriate – what we need is imaginative but co‐ordinated system development and deployment.

c) Arrangements had been made for ongoing maintenance and system support, training

and training materials and, possibly, user support, often but not necessarily with the originating unit. (The centre or even another trials unit might take on part or all of this work).

63 There is therefore an important role for the centre in guaranteeing the long term safety and

utility of any decision to use a community based module. Such modules would not necessarily be free, the ongoing costs of the services above would need to be covered, and there might also be some profit for the originating unit to act as an incentive for development. The modules should be at least competitive with those from the commercial sector, however, and very possibly easier to make conform to standards as they develop further.

64 In some cases the centre could seek to bring together the development effort of teams with existing expertise in the same area, to make the development process more efficient and to further increase the safety and longevity of the final product.

65 There is no reason of course why the various types of collaboration should be confined to academic units in the UK – other units abroad, pharmaceutical companies and system vendors themselves are all potential partners.


Page | 15

66 Implementing modularity will therefore involve a mixture of liaison with vendors and project co‐ordination, albeit of a relatively ‘hands off’ kind. Neither is particularly technically difficult or expensive, though some considerable political skills may be involved, and the process emerges as a natural consequence of introducing system and data standards.

Implementing Support Structures

67 It has already been stressed that a standards based approach will require some central structures and staff, albeit at relatively modest levels, to provide effective support to the introduction of standards. The list below indicates the main functions that will be required, together with an estimate of the input required.

a) Obtaining additional information: Obtaining more comprehensive and detailed information on staffing, funding patterns and costs; on development activity; on views and specific issues; carrying out detailed costings and planning of specific projects. Co‐developing systems to maintain accurate data and information flow – 0.3 fte

b) Supporting system standards: Regular standard review; discussing, describing,

explaining and exemplifying system standards in different units; collecting feedback on use; collating and presenting evidence and feedback; liaising with inspecting bodies / funders; involvement in similar initiatives (e.g. ECRIN WP10) – 0.2 fte

c) Supporting data standards (general): Extensive involvement in a wide range of

standard development groups, including key co‐ordinating involvement in a few; developing, testing and evaluating XML schemas; writing reports and presentations; evaluating standards and their use – 0.3 fte

d) Developing / maintaining data standard tools: identifying potential tools to support

standard introduction; project management of tool development; liaising with staff involved in central repositories; training and developing training materials – 0.5 fte

e) Running / helping to run central data repositories: In particular the metadata item

repository, by preference as part of a collaborative exercise; also collaboration with other CfH / NIHR teams in terms of contact / organisation management – 2.0 fte

f) Support of standards implementation: explaining and discussing data standards in

different trials units; multiple visits and detailed ‘consultancy’ input and guidance in some specific cases – 1.0 fte

g) Supporting modularity: Liaising with vendors; identifying development activity in units;

encouraging information exchange and joint activities; liaising with other potential collaborators; setting up service agreements; project monitoring and management – 0.6 fte


Page | 16

h) Evaluating: Planning, summarising and comparing costs, monitoring, evaluating, reporting and presenting outcomes, on the standards project as a whole and specific components within it – 0.1 fte

68 The total is 5 fte, though not all of this would be needed straightaway. Some of this could

usefully be supplied as purchased consultancy, e.g. from trials unit staff with particular expertise, though it is suggested the bulk is probably best provided as full time input. Some clerical and administrative support would also be required.

69 How such input might be funded and where it might be housed (if, given the roaming nature of large parts of the input, these roles actually need a fixed physical home) is a question for any ultimate funder(s) to consider. The most obvious ‘home’ for much of this work would seem to be the NIHR, and some aspects of the work could and should be quite tightly integrated with ongoing NIHR developments, e.g. in developing the longitudinal research record, the portal and identity management systems. The NIHR would also seem the best organisation to locate the servers and web sites / services that will be required, as it already has a developing infrastructure in this area.

70 Even within the public sector, however, there is a variety of other potential contributors: CfH projects, most obviously the RCP, could also benefit considerably from an effective trials unit standards programme, and there are also the other home nations to consider (for instance the Scottish Office might consider this a better investment than a resurrected Periscope project).

71 The detailed costings of this input, as well as the benefits it would bring, are considered in more detail in the Costs and Benefits section below.

Governance

72 Finally, it is suggested that the DIMS board needs to consider itself. If the project continues then an oversight board, representing all stakeholders, will still be required. (DIMS might be better replaced by a Standards and Information Management Systems or SIMS project, if only for the PR value).

73 Depending on the detailed nature of the project and the groups involved, and especially those funding the project, the board might need reconstitution. In particular, it is suggested that representatives from the other home nations be invited to join the board (whatever the formal funding situation) either as full members or with observer status, because the further any standards project reaches the more effective it becomes. The board is also asked to consider adding an ABPI representative (i.e. over and above the existing representation from pharma), to encourage as much involvement and collaboration from the pharmaceutical industry as possible.

Costs and Benefits


Page | 17

74 The direct costs of the proposals are in the 5 fte input listed above, any clerical / administrative support required, equipment and travel costs, and – in specific cases – the money required to purchase development effort and ongoing support, e.g. for tools to support standards, especially where that comes from outside the immediate trials community.

75 In the first year it is suggested that only a couple of full time staff would be required, plus perhaps some consultancy input from others, and that equipment and other non staff costs would be small. It is suggested that £300K would be sufficient. For the next three years a budget of £500K is estimated, enabling the full complement of staff to be employed (some of which might be consultancy), equipment purchased and systems development supported. After that it is envisaged that some of the immediate need should decrease, as systems start to come on line and mature, so an annual budget of £400K is estimated.

76 These are obviously round figure estimates, and are designed to give substantial ‘head room’ for unanticipated costs. If the proposals are agreed in principle more detailed work would probably be needed to further specify costs. Some costs, e.g. for accommodation, would also depend on decisions about where (or whether) a base was located.

77 It is stressed that not all, or even the main, benefits that would accrue from these costs and the strategy they support are financial. The proposed programme has at its heart the goals of improving the scientific value of the research data, and of improving the general quality of information systems. Nevertheless, because of the potential reduction in duplication of effort by IS staff, particularly in but not restricted to development work, as standards and standard modules become available, substantial financial benefits are also realisable.

78 Table 1 estimates total IS costs in the current (40) registered trials units over the next ten years, and then models the potential savings to be had, if central co‐ordination and support of development is able to reduce the current duplication of effort, using three different scenarios. Details of how the model was constructed are:

a) Current estimated IT costs are used as the baseline. For simplicity these costs have been kept constant over time, i.e. there is no automatic lift for inflation. As explained in more detail in the Consultation Results paper, this baseline figure is made up an estimated 149 IT staff (in registered units), each costing, with on costs, ~£42K.

b) According to Howard et al2 there is a current shortfall in core funded IT posts (83 required – 64 in post). These posts and the associated funding (i.e. 19 * £42K) have been added gradually over years 2 – 8 (2010 to 2016). This seems more realistic than a short term correction given the current economic climate.

c) A ‘quality shortfall’ allowance has also been added, of 15% of current costs, as an estimate of the additional IT funding required to bring all registered units into GCP

2Howard H, Brown J, Meadows H (On behalf of the UKCRC Clinical Trials Units Oversight Group) Review of current core capacity of UKCRC Registered Clinical Trials Units and future resource requirements for the development and delivery of academic‐led, late phase, randomised controlled trials across the UK, November 2008; p 25


Page | 18

compliance with respect to their information systems. This relates to increased validation and documentation activities as much as it does to introducing more sophisticated systems. For some units the additional funding required may be minimal, for others relatively substantial. Again the move to higher quality compliant systems is seen as taking place gradually over years 2 – 8.

d) The two lines of figures added for growth reflect the anticipated increase in the numbers of trials, as described by Howard et al. The first line is for core funded posts only, and is derived as 25/164 (i.e. about 15%) of the average total additional costs given by Howard. (The ratio, which seems a reasonable estimate, is derived from the cost breakdowns in the Howard paper).

e) Because of the recession and its possible effects on research funding, especially by charities, it is not clear if the rapid increase in trials envisaged by Howard and her colleagues is now going to take place. To err on the side of caution, the expansion in trial activity has therefore been ‘right shifted’ by two years, i.e. her Year 1 figure is Year 3 in this table, and years 1 and 2 gradually step up to the year 3 level.

f) The second line shows the growth related figure for non core funded staff. It is derived from the core figure and assumes a 45/55 split between core and non‐core IT costs. This is almost certainly an under‐estimate of the actual proportion of non core costs, but helps to keep the total cost estimates reasonably conservative.

g) The various cost lines listed above are summed to give the total IT costs for the registered units, which are then broken down into 40% development, 60% other work.

Registered Units 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

Costs Current 6.30 6.30 6.30 6.30 6.30 6.30 6.30 6.30 6.30 6.30

13% Current Shortfall 0.00 0.12 0.23 0.35 0.47 0.59 0.70 0.82 0.82 0.82

15% Quality Shortfall 0.00 0.14 0.27 0.41 0.54 0.68 0.81 0.95 0.95 0.95

Growth, RUs Core 0.21 0.42 0.62 1.83 2.45 2.74 2.73 2.74 2.94 2.99

Growth, RUs NonCore 0.26 0.51 0.75 2.24 3.00 3.35 3.34 3.35 3.60 3.65

Total 6.77 7.49 8.18 11.12 12.76 13.66 13.88 14.16 14.60 14.70

Development 2.71 2.99 3.27 4.45 5.10 5.46 5.55 5.66 5.84 5.88

Other work 4.06 4.49 4.91 6.67 7.66 8.19 8.33 8.50 8.76 8.82

With DIMS…

Central Support 0.30 0.50 0.50 0.50 0.40 0.40 0.40 0.40 0.40 0.40

Potential Savings 20% Dev. Savings 0.00 0.00 ‐0.34 ‐0.68 ‐1.02 ‐1.09 ‐1.11 ‐1.13 ‐1.17 ‐1.18

30% DS + 5% Other 0.00 0.00 ‐0.64 ‐1.28 ‐1.91 ‐2.05 ‐2.08 ‐2.12 ‐2.19 ‐2.21

40% DS + 10% Other 0.00 0.00 ‐0.94 ‐1.87 ‐2.81 ‐3.00 ‐3.05 ‐3.12 ‐3.21 ‐3.23

Revised Totals

20% Dev. Savings 7.07 7.99 8.34 10.94 12.14 12.96 13.17 13.43 13.83 13.93

30% DS + 5% Other 7.07 7.99 8.04 10.34 11.25 12.01 12.19 12.44 12.81 12.90

40% DS + 10% Other 7.07 7.99 7.74 9.75 10.35 11.05 11.22 11.45 11.79 11.87

ROI


Page | 19

20% Dev. Savings 0.30 0.80 0.96 0.78 0.16 ‐0.53 ‐1.24 ‐1.98 ‐2.75 ‐3.52

30% DS + 5% Other 0.30 0.80 0.66 ‐0.11 ‐1.63 ‐3.28 ‐4.96 ‐6.68 ‐8.47 ‐10.28

40% DS + 10% Other 0.30 0.80 0.36 ‐1.01 ‐3.42 ‐6.02 ‐8.67 ‐11.39 ‐14.20 ‐17.04

Table 1: Postulated IS Costs, Registered trials units, 2009 – 18

h) The input costs for central support of the proposed DIMS strategy are then added, £300K in year 1, £500K for the next 3 years, then £400K p.a. for the remaining six years, reflecting a likely time pattern of input.

i) Three scenarios are then considered, as possible financial consequences of the

investment provided.

1 The first has a gradual reduction in the pro‐rata development (and associated design, documentation and validation) effort required, in years 3, 4 and 5, to 7%, 14% and 20% of that estimated, staying at the 20% level thereafter

2 The second has the same time pattern but is based on an eventual 30% reduction in development effort. A reduction of this magnitude ought to be possible given the amount of duplication currently occurring, and could act as one of the 5 year targets for the initiative. In addition, because development work is not the only aspect of information system activity that could be made more efficient by a standards based, modular approach (providing training and support materials, data extraction and reporting techniques are other examples) there is an additional 5% reduction in all the ‘other’ work involved.

3 The third scenario is a more ambitious, but still attainable, version of the second, with reductions in pro‐rata development costs of 40% and a 10% reduction in ‘other’ IT activity

j) These three scenarios give the revised total annual costings, i.e. adding the trials units

IT costs and the central support costs, but subtracting any identified savings. The three ROI rows provide the same data but as accumulated costs / savings. It can be seen that there is a positive return on investment in years 4 or 6 depending upon the scenario selected.

79 The next table shows a similar estimate for all trials units. The main differences is that the core

and non‐core costs for the registered units are combined into one line, and are then used as the basis of an additional total cost for the non registered units, based on a 65 / 35 split (again this is a conservative estimate, as registered units are currently involved with 55% of trials, though that figure is anticipated to rise).

All Units 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018Costs

Current 8.40 8.40 8.40 8.40 8.40 8.40 8.40 8.40 8.40 8.40

13% Current Shortfall 0.00 0.12 0.23 0.35 0.47 0.59 0.70 0.82 0.82 0.82

20% Quality Shortfall 0.00 0.24 0.48 0.72 0.96 1.20 1.44 1.68 1.68 1.68


Page | 20

Growth, Reg. Units 0.47 0.93 1.37 4.07 5.45 6.10 6.06 6.10 6.54 6.64

Growth NonReg.Units 0.25 0.50 0.74 2.19 2.94 3.28 3.27 3.28 3.52 3.58

Total 9.12 10.19 11.22 15.72 18.22 19.57 19.87 20.28 20.96 21.11

Development 3.65 4.08 4.49 6.29 7.29 7.83 7.95 8.11 8.38 8.45

Other work 5.47 6.12 6.73 9.43 10.93 11.74 11.92 12.17 12.57 12.67

With DIMS…

Central Support 0.30 0.50 0.50 0.50 0.40 0.40 0.40 0.40 0.40 0.40

Potential Savings

20% Dev. Savings 0.00 0.00 ‐0.49 ‐0.97 ‐1.46 ‐1.57 ‐1.59 ‐1.62 ‐1.68 ‐1.69

30% DS + 5% Other 0.00 0.00 ‐0.91 ‐1.82 ‐2.73 ‐2.93 ‐2.98 ‐3.04 ‐3.14 ‐3.17

40% DS + 10% Other 0.00 0.00 ‐1.34 ‐2.67 ‐4.01 ‐4.30 ‐4.37 ‐4.46 ‐4.61 ‐4.65

Revised Totals

20% Dev. Savings 9.42 10.69 11.24 15.25 17.16 18.40 18.68 19.06 19.68 19.82

30% DS + 5% Other 9.42 10.69 10.81 14.40 15.89 17.03 17.29 17.64 18.21 18.35

40% DS + 10% Other 9.42 10.69 10.39 13.55 14.61 15.66 15.90 16.22 16.75 16.87

ROI

20% Dev. Savings 0.30 0.80 0.81 0.34 ‐0.71 ‐1.88 ‐3.07 ‐4.29 ‐5.57 ‐6.86

30% DS + 5% Other 0.30 0.80 0.39 ‐0.93 ‐3.27 ‐5.80 ‐8.38 ‐11.02 ‐13.77 ‐16.53

40% DS + 10% Other 0.30 0.80 ‐0.04 ‐2.21 ‐5.82 ‐9.72 ‐13.69 ‐17.75 ‐21.96 ‐26.21

Table 2: Postulated IS Costs, All trials units, 2009 – 18

80 To reflect the fact that more input is likely to be needed to raise many non registered units to the required operational IS standards (including some units that are currently seeking to become registered) the ‘quality shortfall’ figure has been raised to 20%.

81 The pattern of possible costs, for all trials units, is illustrated by Figure 1, which graphs the total costs for the various scenarios. It can be seen that the central investment causes a small net increase in costs for the first three years, as one would expect, but that in years 4 and 5 the curves dip down below that representing the ‘no change’ line, and then climb much more slowly than the ‘no change’ line. The figures at the end of each line represent the extrapolated total cost of trials units IS in 2018.


Page | 21

Figure 1: Postulated IT Costs, All trials units, 2008 – 18, showing the effect of various levels of savings in development and other work gained from a standards based modular approach.

82 Because of the inevitable uncertainties about the base data and the extrapolations made, IT costs in the model were kept deliberately conservative, whilst the funding requirements were, if anything, exaggerated. Nevertheless, even with these constraints, the standards based approach can be seen to deliver substantial and increasing savings after 4 or 5 years. Such savings (it is stressed once again) are in addition to the very considerable gains in quality and scientific value that are the other goals of the proposed strategy.


Page | 22

Conclusion

83 The DIMS project offers, for the first time, the opportunity to establish a clear national strategy for information systems in UK academic trials units. It is important that we take that opportunity in the most imaginative way possible, standing back from servers and keyboards, queries and code, to examine how clinical research, in the trials units and beyond, is best served.

84 Good research is based on good quality data, however that might be collected, now or in the future. The data is best organised and most efficiently described using appropriate standards. The technology therefore needs to support the data and the standards, demonstrating that support by conformance to predefined criteria.

85 Technology that is standards based no longer locks data away inside itself – it can exchange information with other systems, at different stages of processing, because they share the same structures for organising the data and because they agree upon its meaning. It no longer needs to be built as a single system, it becomes instead a collection of functional components tied together by the all pervasive standards.

86 Debates about systems can then move on. Ideally, we assemble what we want, when we want, and we don’t care whence it came as long as it works. Standards do not just break up systems, they break up the sterile arguments that surround them as well.

87 The strategy outlined in this paper would allow us to move down this route and develop higher quality, more consistent tools to support research. The project board are asked to endorse it.


Page | 23

Document History

Version Date Issued

Brief Summary of Change Owner’s Name/Signature

0.1 18/03/2009 Initial version, distributed for comments

Steve Canham

0.2 28/03/2009 Descriptive paragraphs on Periscope and Cancer Grid received from Sharon Kean (Glasgow University) and Professor Jim Davies (Oxford University) and incorporated.

Steve Canham

0.3 30/03/2009 Various amendments to the text Steve Canham After feedback from Will Crocombe and Jim Charvill

0.4 01/04/2009 Costing models, tables and graph included to illustrate potential financial benefit. Costs and Benefits section added.

Steve Canham After feedback from Richard Farr and Jim Charvill

0.5 23/05/2009 Minor amendments Steve Canham following DIMS board meeting, UKCRC CTU OG


Page | 24

This is an NIHR controlled document. On receipt of a new version, please destroy all previous versions.

Documents

Trials Units Information Systems General Proposals … · Trials Units Information Systems – General Proposals ... Introduction ... and dissemination of software development projects,