Central dimensions of clinical practice evaluation: efficiency, appropriateness and effectiveness - I

Journal of Evaluation in Clinical Practice, 2, 1, 13-27

Cent ra I d i mens ions of cl in ica I practice eva I ua t io n : efficiency, appropriateness and effectiveness - I Decian O’Neill MPH FAFPHM MFPHM‘, Andrew Miles BSc(Hons) MSc MPhil PhD2 and Andreas Polychronis MB BCh3 ‘Director of Health Improvement, New South Wales Health Department, Sydney, Australia *Deputy Director, Centre for the Advancement of Clinical Practice, European Institute of Health and Medical Sciences, University of Surrey, UK 3House Physician, Department of Medicine, Llandough Hospital, Cardiff, UK

Correspondence Abstract Dr D. ONeill Director of Health Improvement New South Wales Health Department Locked Mail Bag 961

Successful audit of clinical practice focuses upon the systematic investigation of key aspects of the everyday work of busy clinicians. We contend that the nature and quality of local clinical practice can be characterized by critical

Sydney NSW 2059 Australia

Keywords: appropriateness, audit, effectiveness, efficiency, evaluation, evidence-based care, utilization

Accepted for publication: 16 February 1996

examinations of the effectiveness and appropriateness of practice and the efficiency with which effective, appropriate clinical care is delivered to patients. When such a baseline has been established, it becomes possible to compare and contrast characterized local practice with so-called ‘evidence- based’ practice and agree changes aimed at narrowing the discrepancy between the two. The nature of such changes can be described and their implementation into practice studied, with subsequent quantitative measurement and qualitative description of the resulting benefits to patients. A proper understanding of the concepts of efficiency, appropriateness and effectiveness in clinical care is clearly fundamental to the successful design and applications of methodologies aimed at securing measurable improvements in the quality of patient care. In this first of two articles we examine the concepts of efficiency and appropriateness in clinical practice, with particular emphasis on cost-effectiveness and utilization review. The clinical effectiveness of health care intervention is treated in detail within the second paper, to be published within Volume 2 Number 2 of the Journal of Evaluation in Clinical Practice (Miles et al. 19960.

Introduction

Relman’s description of three revolutions in health care provide a chronicle of change on the part of the various players in health care (Relman 1988) First came the era of expansion where medicine provided the leads (the ‘what ifs’), technology and biomedical research took up the challenges, and government and managers responded through the provision of resources. This post-war period of enthusiasm was soon to be followed by the era of fiscal constraint. The health care slice of the public spending pie was growing. Treasurers one after another, depending on

the depth of their pockets, signalled the limits of their largesse to their governments. Governments and management then provided their own leads by applying financial brakes with medicine, research, technology and the consumer having to tighten their belts. As health care is a dynamic system, this situation was unlikely to persist unchanged. And so we have moved into the third era, that of evaluation where all the players, including consumers, have a pre- rogative to provide the leads.

In the past century, life expectancy at birth has risen by about 50% (Murphy & Charlton 1995). Much of that increase has resulted from improvements in

0 1996 Blackwell Science 13

D. O’Neill eta/.

overall living standards rather than from any specific health care strategies. How much, if any, direct con- tribution health care makes to average life expectancy is not clear, but it is unlikely to represent much more than a year of additional life expectancy.

In the UK, the amount spent annually on NHS- provided health care is heading towards €50 billion (Central Statistical Office 1994). In addition, there is increasing expenditure in the private sector. The proportions of elderly and very elderly people in the population continue to increase, as does the potential to treat more conditions and to devise better ways of packaging care. This creates a powerful stimulus in favour of increasing the resources available for health care consumption. In some Western countries this increase appears to have been unstoppable. Health care expenditure in the USA has almost doubled over the past 25 years. In France, the proportion of GDP consumed by health care has risen by nearly 50% in a decade. In many other Western countries, however, there has been little change over the past 10 or 15 years in the proportion of GDP allocated to health care (Fenn & Mayhew 1991).

Just as the resources allocated to health care com- pete against other demands on the public purse, so too there are competing priorities to be found within the envelope of the health allocation. With the range and depth of useful health interventions increasing almost daily, the next logical steps is to evaluate what is being done in order to extract the maximum benefit from the resources available within the envelope. In practice this means identifying where need, access, efficiency and effectiveness are each lowest and highest. By developing an understanding of health care delivery at this level it becomes increasingly possible to justify resource shifts on the basis of robust evidence.

Within the maelstrom of complex health care provision this philosophy of change based on evaluation requires certain reference points from which to embark. These reference points include the following. 0 Identification of likely demand patterns through

the assessment of need and access to care. 0 Identification of differences in care through the

study of variations. 0 Identification of efficiency of care through studies

of utilization cost benefit and cost utility analysis. 0 Identification of effective care through reviewing

the evidence base of care, developing and dis-

seminating best practice consensus and standards, auditing clinical activity, testing clinical appropriateness, assessing health outcome and attri- buting change to intervention or otherwise.

In the contemporary NHS, the responsibility for health strategy lies with the purchasers of health care, and virtually all major developments should now be based on a detailed understanding of need in the community .

Variations in the use of health care have been reported as far back as 1850 (Jarvis). They have been reported within countries (Wennberg & Gittlesohn 1982) and across countries (Pearson et al. 1968), and the reader is referred to Krakauer et al. (1 995) for a detailed, systematic assessment of variations in medical practices and their outcomes. The underlying factors have been studied in depth and conveniently broken down into three major categories: need and demand factors, supply factors and clinical factors relating to decision making and uncertainty. The use of variation studies in any field or process provides a point or points from whence to embark on discussion and further analysis of the various factors involved.

Efficiency of care delivery is of prime importance in any health system, ‘value for money’ being the term commonly used in the NHS. No successful financial system runs without controls, and governments have a responsibility to see that maximum return is achieved for every taxpayer’s pound spent on health. In a system which spends f50 billion, it is essential to have checks and balances in place. An example of such a control in the system in the Efficiency Index (National Health Services Management Executive 1991). This is a weighted index, based primarily on the numbers of complete consultant episode, for expressing activity increases at Acute Trust Units. It is based on an assumption that, through efficient operation and economies of scale, a marginal increase in annual productivity should be expected from every unit, before any other arrangements for specific adjust- ments to output are agreed.

More sophisticated ways of demonstrating efficiency in clinical services have been described. These include methods taken from accounting models which create algorithms using ratios of data elements which are produced for routine financial statements, and methods taken from the manufacturing industry which look at the technical efficiency of decision-

14 0 1996 Blackwell Science, Journal ol Evaluation in Clinical Pracrice. 2, 1, 13-27

Central dimensions of clinical practice evaluation

making units in their use of similar inputs for the production of similar outputs (Ehreth 1994). Another example is the use by Family Health Service Autho- rities (FHSAs) of prescribing allocations for fund- holders and indicative prescribing plans for non- fundholder general practitioners. These involve financial and therapeutic target settings for each practitioner.

Along with the money spent by or on behalf of the patient on health, another cost factor is time. This manifests itself in issues such as the timing of a preventive strategy, the person-time spent on a wait- ing list, accumulated person years of disability, and potential person years of life lost to premature mortality. Strategies aimed at modifying the time spent queuing for care, or in reducing premature mortality, are in one respect similar to efficiency indices and indicative prescribing budgets. They focus on the parts of the system that are the most accessible and analysable. To a certain degree, many of these methods are confined to one side of the equation. This is the ‘inputs’ side. On the other side of the equation there are other levers which historically have been less frequently the focus of initiatives or strategies for modifying health and health care. These include both ‘outputs’ and ‘outcomes’. The term ‘output’ is used here to describe the type of care provided; for example, care of coronary heart disease, learning disability, cancer and so on, and the way in which it is provided; for example, outpatient care, inpatient care, investigational process and specific therapies. The analysis of outputs includes consideration of alternative ways of providing such care and their relative costs and benefits. In evaluating outcome we are seeking to identify the change in health status expected, achieved or not achieved, in the short and long terms, and to what intervention or other aspect of care the outcome can or cannot be attributed.

In order to make some sense of this complex world of decision-making processes, expectancy and results, it is helpful to segment it into different parts. This in itself is artificial, as the area of interest is a continuum rather than a series of independent steps or variables. However, for practical purposes we will look sepa- rately, where possible, at efficient and appropriate utilization of health care on the one hand, and the provision of eflective health care on the other.

Utilization of health care

The comparison of different ways of achieving similar outputs provides information which assists in decisions on resource allocation. There is growing interest in analysing the utilization of services. A number of policies and changing trends have stimulated this interest, and these include the institution of a managed market ,in the NHS, the development of management within clinical services, the move towards a primary care focus in health care and the establishment of clinical audit (Miles et al. 1996a,b). Accom- panying this wave of change have been trends of change in the utilization of care. Shorter inpatient stays and same-day care are common worldwide. A specific trend being reported by some acute trusts in the NHS is an increasing proportion of emergency admissions (Hobbs 1995).

Utilization can be studied at any level in the system and in any sector, although the acute sector has been the centre for many developments. The principles are, in general, transferable across sectors. The most straightforward type of care utilization study is that of categorization and role delineation. This is based on parameters such as type and extent of staffing, support teaching and research. Levels of care are defined which range from the base category, in which ambulatory outpatient care is provided by medical practitioners, registered nurses, or allied health professionals, up to the most complex category, in which 24-hour specialist medical nursing and allied health professional cover, sophisticated investigational support, specialized teaching and research, regional and supra-regional referral services may be involved.

The services provided by any unit or trust can be broken down into sub-units (e.g. specialities) and analysed. The need for subunit breakdown is obvious in that one trust of a teaching hospital might have, for example, the supra-regional cardiological services, but no psychiatry or paediatrics. In some services, acute and community care are amalgamated, but in others they are mutually exclusive. By looking at the levels of service available and the risk-adjusted case mix being dealt with, service models of similar dimensions can be identified. These kinds of role definition models provide us with information on the levels of care available and their use. They do not provide insight

Q 1996 Blackwell Science, Journal ol Evaluation in Clinical Practlce, 2, 1, 13-27 15

D. O’Neill eta/.

into whether the care is needed or whether the care being used is at the appropriate level. Neither do they demonstrate whether the outcomes achieved, or not achieved, were those intended, or whether the outcome was attributable to the care provided.

Utilization review

Background

Utilization review is the review of the patient’s care through application of defined criteria and expert opinion. This is usually done through medical record reviews (Payne 1987), but other methods, such as patient or staff interviews, have also been described (Anderson et al. 1987). Although many people in different parts of the world have researched ways of evaluating the care used, the major impetus for development in this area came from US Government policy in response to rising health care costs when it legislated for such activity in the case of medicare reimbursed patients (U.S. Department of Health, Education and Welfare 1974). As the system in that country has developed, it has evolved into a culture of external review exercised through peer group organizations, as required by statute (Professional Standards Review Organizations; PSRO) and payer-appointed peer reviews (as private insurers have followed the medicare lead).

The main type of review outlined in the PSRO system was retrospective and concurrent review. It was envisaged that, over time, sets of criteria would be developed against which inpatient admissions and subsequent days of stay could be checked. Batches of cases would initially be screened by trained nurse reviewers. Those cases found not to have met any criteria at the initial screen would then be rechecked by a clinician reviewer and, where further indicated, by a peer review committee. The two-phase approach was aimed at (i) assuring, at admissions review, that a hospital level of care is medically necessary, and (ii) placing a length-of-stay checkpoint on certified admissions for continued stay review. Initially introduced as a retrospective review of cases after the hospital stay, this gave rise to controversy over ‘retrospective denials of payment’, and led to concurrent review of patients as they were being admitted. This process has evolved into what is commonly known as utilization review.

I wo otner caregones 01 review were also proposea. Medical care evaluation studies. This technique was envisaged as a method for emphasizing quality improvement through continuing medical education. It was aimed at the specific correction of identified problems through a structured review process including medical audit pathways, education processes and repeated review. It represents traditional clinical audit (Miles et al. 1996a,b). Profile analysis. This activity involves retrospective review of aggregated patient care data subjected to various pattern analyses, such as lengths of stay, resource usage, adjusted risk, etc., according to factors such as casemix, clinician, speciality and hospital.

-,- .. . , . .

Types of utilization review

Different types of utilization review have been described (Payne 1987). They fall into four main categories.

Implicit criteria. A clinician reviewer directly applies his or her own judgement to the quality and or appropriateness of the care provided. The validity therefore depends entirely on the knowledge, skill and judgement of the reviewer. Length of stay ( L O S ) profiles. The title is self- explanatory. Here case-mix structured profiles are used. They are generally designed to reflect mean clinical LOS for region or district and mean peer LOS for unit or hospital, and to provide exception reporting for cases outside 2 standard deviations from the mean. Explicit criteria - diagnosis-independent. This is a criteria audit. Explicit criteria are used which identify appropriate acute inpatient levels of care. Standard criteria sets are used which reflect specific medical and nursing services (e.g. operation same day) or non-specific conditions (e.g. elec- trolyte imbalance), thereby giving an indication of the level of service needed or the severity of the illness suffered. Patient medical records are screened. The finding of one or more criteria in the patient’s record indicates an appropriate inpatient admission. Finding more indicates the converse. The findings are subject to further refinement through clinical review. They are used for screen-

16 0 1996 Blackwell Science, Journal o l Evaluation in Clinical Practice. 2 , 1 , 13-27


0

ing activity through recorded patient information (the medical record), in conjunction with a review by clinicians of incidences where no criteria are met. Explicit criteria - diagnosis specific. This is a diagnostic criteria audit. Individual guidelines are established for specific categories of patients and conditions. The guidelines are complex instruments whose establishment represents a substantial workload based on evidence and expert consensus relating to the appropriateness of the care provided for a particular set of clinical circumstances. The review process is structured and complex in itself.

Methods of use

All of these types of utilization review instruments have been used at various times in various environ- ments.

Implicit criteria When researchers first started to examine the utilization of clinical services, the curse of implicit criteria was common. In these studies, the decision as to whether or not a referral, an attendance, an admission or a day of stay is appropriate is made by the reviewer. The validity of such methods, therefore, depends entirely on the skills, judgement and knowledge of the reviewer (Donabedian 1982). The inherent weakness in this approach is the incon- sistency that has been demonstrated between clinician reviewers. Agreement rates using these techniques have been described as being at or near the level expected by chance alone (Sanazaro 1980). Methods of overcoming this weakness have included stratifi- cation of cases into subgroups based on casemix, lengths of stay and other such parameters; the inclu- sion and weighting of a range of opinion groups such as patients, carers, general practitioners, nursing staff, junior doctors and consultants: the prior development of consensus opinion for a range of condition states through Delphi-type techniques with incorporation into the decisions on appropriateness. Each of these modifications was found to have its own cost. It has been common experience that, the more the instruments are enhanced, the more cumbersome they become to use. This difficulty has led to the development (described below) of explicit criteria-type

instruments, but their use has been mainly confined, so far, to acute inpatient services or to specific conditions or procedures. In general, studies investigating referral pathways or the use of ambulatory care or non-acute care must still rely on this type of instrument.

Length of stay profiles Of particular use for examin- ing the margins of a tranche of clinical activity are ‘length of stay profiles’ and other similar techniques which focus on parameters directly related to patterns of service consumption. The aim of this type of exercise is to create utilization proj2es for specific areas of activity such as whole hospitals, speciality units, dis- crete wards or procedure units, or individual clinician workloads. Such profiles are based on case mixes and lengths of stay for different clinician groups. There is no limit, however, to the range of variables which can be used to create a profile of activity. The profile is presented in terms of the distribution of overall activity. The cases falling outside (say) 2 standard deviations for a particular variable, for example length of stay, are then examined in further detail for underlying factors which may (or may not) be redu- cible or avoidable.

Depending on the variables used, the profile can be tailored for particular purposes. An example might be the profile of chest X-ray usage of admitting house officers. The main limitation with this type of exercise is that it focuses on what is marginal and provides little or no information about the bulk of activity. The technique can, however, be particularly useful for studying high cost, high risk, high dependence or other aspects of a clinical service. Another use could be monitoring guideline usage (e.g. imaging guidelines) (see Miles et al. 1996c-e).

Explicit criteria - diagnosis-independent These instruments remain in common use and have been refined over the years. The use of two such instruments has been widespread in North America, namely the Appropriateness Evaluation Protocol (AEP) (Gertmann & Restuccia 1981) and the Intensity, Severity, Discharge screens-A (ISD-A) (InterQual Inc. 1987). This type of utilization review is relatively simple and inexpensive and is being found to have potential for effective application in the NHS, particularly at a local clinical management level, with an

Q 1996 Blackwell Science, Journal of Eveluafion in Clinical Practice, 2, 1, 13-27 17

D. O’Neill eta/.

increasing number of surveys being reported from around the country. Diagnosis-independent (explicit criteria) instruments have been shown to be quite sensitive in determining the levels of efficient utilization of acute services. One such tool, the AEP (and its paediatric modification PAEP), has demonstrated sensitivities of 90% or more (Gertmann & Restuccia 1981; Kempner 1988). However, when used in the PSRO programme, some of the sensitivity and speci- ficity of these instruments appears to be lost (Dippe et al. 1989).

It was noted by the developers of the AEP that specific agreement rates for inappropriateness corre- lated with low familiar reviews were with the institution in question and its clinical capabilities (Restuccia 1982). A good case therefore exists for use of these instruments as part of provider clinical audit, rather than as a form of external review such as that required by payers in the US. As will be seen below, evidence of difficulties associated with remote use has already been found in a UK experiment.

The instrument themselves consist of lists of criteria which reflect need for care, severity and dependence. Different criteria lists are used for evaluation of admission days and days of stay. In the standard method of use the instruments are applied by screening inpatient medical records against the criteria. Those records where no criteria were met are then subjected to a further review by a panel of clinicians. Where a case is considered appropriate, although not having met any criteria, the reviewing clinicians operate an override to include it with those deemed appropriate. For the remaining cases the reasons for inappropriateness are ascertained and recorded. The result of a survey provides an estimate of appropriate utilization and a profile of the underlying reasons for inappropriate utilization. Different modes of use are as 0

0

_ _ - follows. Retrospective. A sample of inpatient records from a period in the recent past are examined. Any conclusions drawn from the findings are applied to current management practice in order to demonstrate change at future review. Concurrent. A review is conducted on the records of current inpatients. As barriers to appropriateness are recognized, the information is fed back to the responsible clinicians in order to stimulate change ‘concurrently’.

0 Prospective. Instead of reviewing the status of the patient on or after admission, the criteria are used for pre-admission screening and for continued stay screening at a certain point after admission. This method of use is intended to inform decision making in advance.

Explicit criteria - diagnosis-specific The fourth type of review instrument has been developed through the incorporation of evidence-and consensus-based clinical guidelines into criteria sets. These criteria sets outline the specific circumstances where the use of a particular clinical procedure or line of clinical management is (i) clearly appropriate, (ii) of doubtful appropriateness or (ii) clearly inappropriate. The criteria are used as a clinical audit tool for estimating the levels of appropriate use of a specific intervention across a patient population.

Experience and developments in utilization management and review

United States of America

Utilization management and review are widely prac- tised. The driving philosophy behind their inception was to reduce the costs involved in acute care provision. With the diagnosis-independent instruments there have been attempts to shift to concurrent review, in response to reported greater behaviour modification with this than with the retrospective mode (Restuccia 1982). However, the practical use of these instruments does not sit well with the type of review required of Peer Review Organization (PRO) pro- grammes.

Recent developments have seen increasing use of diagnosis-specific (explicit criteria) instruments. Thus the utilization management brief has been widened, moving from a simple efficiency focus to include effectiveness. Certain insurers and Health Main- tenance Organizations have introduced proactive management of admissions and continued stay. Using computer-based checklists, selected admissions are ‘screened’ for appropriateness. The application of computer-based diagnosis-independent instruments as a means of developing managed care has also been described (Utilization Managements Associates 1993).

18 Q 1996 Blackwell Science, Journal of Evaluation in Clinical Practice. 2, 1, 13-27


United Kingdom

In the UK, the results of studies of the appropriateness of acute hospital use have been reported inter- mittently for over 30 years.

Implicit criteria instruments A cluster of studies appeared towards the end of the 1950s. The first consisted of medical record reviews and interviews with relatives of patients discharged from a Birming- ham hospital. ‘No diagnostic or therapeutic require- ments at hospital level were reported’ in a quarter of patients (Crombie & Cross 1959). A second, using consultant opinions, reported similar findings for existing inpatients (Forsyth & Logan 1960). A further, larger survey in Birmingham, focusing on admissions and delayed discharges, reported 4.7% admissions, and a further 13.3% of delayed discharges, unneces- sary on medical grounds alone (Mackintosh & Garrat 1961).

All of these studies used implicit clinical opinion in determining the appropriateness or otherwise of a case. Later researchers used different ways to try to overcome the inherent weakness of implicit criteria. Torrance et al. ( 1 972) channelled opinion into general categories of reasons for admission (e.g. consultant opinion required or alternative services unavailable) and introduced basic explicit needs criteria (medical needs, nursing needs, etc.). Rosser (1 976) put forward a method for incorporating consensus clinician opinion derived from local use of ‘Delphi techniques.

In common with similar studies, these were found to be of doubtful validity across different environ- ments, or too cumbersome for practical wider use when heavily structured. They have continued to be reported for specific settings from time to time (Murphy 1977; Farag & Tinker 1985; Coid & Crome 1986; Beech et al. 1987).

Explicit criteria - diagnosis-independent instruments Donaldson et al. (1972) reported a comprehensive explicit criteria instrument at around the same time that the first explicit criteria instruments were appearing in the USA. A report of the development of an audit instrument consisting of nine explicit criteria applied, through interviews with nursing staff, to inpatient days of stay was published by Anderson et af . (1987); this was described as the ‘Bed study instrument’. Several studies have been reported in the

recent past (Harvey et al. 1993; Victor et al. 1993; Coast et al. 1994; Houghton 1995; Victor & Khanoo 1994; Bogg et al. 1995; Lawrence & Edwards 1995; U. Wernicke, London Health Economics Consortium, personal communication), in which the three common instruments were used. Further prospective research into the validity and acceptance of this sort of evaluation of care is being undertaken in some large multicentre studies (Beech et al. 1995; Fenn et al. 1995; H. Smith, Wessex Primary Care Research Net- work, personal communication).

Explicit criteria - diagnosis-specijk instruments The use of diagnosis-specific instruments in the United Kingdom has been limited. Development of instruments as described above took place in Trent Region in the late 1980s (Hampton 1989). A project has recently been established at Lewisham in South Thames Region in which diagnosis-specific instruments for upper gastro-intestinal endoscopy, tonsil- lectomy, grommets, dilatation and curettage, and hysterectomy are being modified and tested. The panels for the modification exercise are made up of two general practitioners, one internal specialist in the field from the hospital, one external specialist from elsewhere, a specialist clinician from an unassociated clinical specialty and a public health physician (K. Stott, Lewisham Hospital, personal communication).

Appropriateness of hospital use in Europe

A European study group was established in 1994, through an EC BIOMED project grant. Included in the group’s objectives are the development of common core instruments for evaluating utilization and the identification of reasons underlying inappropriate utilization (Liberati 1995). Work is currently being conducted on the development of a common Eur- opean instrument for review of diagnosis-independent utilization (explicit criteria). A comprehensive review of the project and activities in six European countries is provided by Liberati et al. (1995).

Focused utilization analysis and proactive management of utilization

By conducting a utilization review, major bamers to appropriate utilization are made evident. Of those

Q 1996 Blackwell Science. Journal ol Evaluation in Clinicel Practice. 2. 1, 13-27 19

D. ONeill eta/.

which fall within the responsibility of the organization’s internal management, some may be addressed through straightforward policy change or other management practice. For other barriers, particularly subtle systematic practices or admissions, it may be desirable to introduce systematic clinical management processes to assist with their identification and with reorientation of practice.

There is a group of quality assurance techniques described as ‘problem orientated approaches’ (Demlo 1983) which focus on specific problems in the system in order to fix them. Recent NHS experience with two such practices has been reported. Adverse event monitoring is the systematic detection, investigation and analysis of events which can identify the provision of poor-quality care (Bennett & Walshe 1990). Patient focused care is a detailed analysis system which looks at care from the patient’s position, and uses a pro- jection of the ‘anticipated recovery pathway’ for each type of patient. This enables the optimal pathways through the various care processes to be mapped. The mapping then reveals the common loci for barriers to good utilization or factors detrimental to maintaining and improving quality (C. Bruce & J. Van Liew, Integrated care: North West Thames Health Authority. Medical management experience from the field. Paper presented at British Association of Medi- cal Managers Autumn Conference, 1992).

Utilization review and the health care system

The Department of Health is strategically set to move the focus of health care from the secondary and tertiary care sectors to the primary care sector (Depart- ment of Health 1994). Shifting of the focus includes shifting of a certain amount of care. This position assumes that there is a significant case-load which is currently dealt with by the secondary and tertiary care sectors but which could be more appropriately managed at an alternative level of care. Such a reorgani- zation would then be expected to result in: 1 a decrease in the overservicing of patients who are inappropriately receiving secondary level care; 2 a decrease in the underservicing of patients who should appropriately be receiving primary and community level care; 3 some potential to reduce the underservicing of patients who should appropriately be receiving sec-

ondary level care but are either in the system but not receiving care or not being referred. The difficulty in translating this theory into practice lies in finding commonly agreed definitions of levels of care at any point in the system. Initial attention should be concentrated at the locus at which there is the strongest potential for influencing movement across levels. In the current situation, many suppo- sedly acute units find themselves responding to demands for services which may not be acute because there is no appropriate supply elsewhere. The need to define what is and what is not acute care is clear.

Systematic analysis of acute admission days, and days of stay, against an agreed set of criteria which indicate acute care can be used to assess provider activity. A study of subgroups which fail to meet the criteria can be used to identify the major reasons why patients are receiving inappropriate care. Some of the reasons found will be amenable to internal managerial action. Other reasons will be identified which relate to barriers to appropriate use arising from the external environment.

A further potential development is to devise criteria sets for other levels of care such as community hospitals, extended care units, hospitals at home and services outside the NHS such as nursing homes. For each of these situations it would be necessary to consider criteria which delineate more than one interface with other levels of care.

The consideration of efficient utilization is only one aspect of evaluation of the appropriateness of heal thcare. Appropriateness must also be considered in more intrinsically clinical terms.

Clinical appropriateness

The clinical appropriateness initiative, as it has been described, was conceived by Brook in 1988 (Chassin el al. 1989) with the principal aim of investigating the contention that approximately 20-30% of common medical and surgical procedures undertaken in the United States were performed in a setting in which the expected risks of the procedure were calculated to be in excess of the anticipated health benefit. As a consequence, approximately one-third of American health care was potentially ‘inappropriate’, and Brook suggested that a clinical appropriateness initiative would represent a powerful tool in reducing the use of

20 0 1996 Blackwell Science, Journal of Evaluation in Clinical Practice. 2, 1, 13-27


health care resources, which was actively contributing to an unsustainable increasing proportion of the US Gross National Product being allocated to health care delivery (Ballard 1994).

Hopkins (1989) has made the point that many clinicians will view examinations of appropriateness of, for example, test and procedure selection as largely representing a ‘cost-cutting exercise’. There is, of course, as Hopkins (1989) has pointed out, enormous scope for saving money by avoiding inappropriate investigation and treatment, but clinicians should note that appropriateness studies can also reveal areas of inappropriate under investigation and treatment (Yi et al. 1989) indicating the need for investment of resources.

Only relatively recently has the word ‘appropriate’ jointed the words ‘effective’, ‘efficient’, ‘outcomes’ and ‘health gain’ in the specialist vocabulary of health care. As a consequence, some confusion is likely to occur in its use. The implied issue is directly related to the suitability or unsuitability of a given approach to treatment for a given patient or patient category. The main criterion is patient benefit and, although it is clear that a potential treatment should be evaluated in terms of risk to the patient, it remains less clear whether considerations of the cost of treatment should occupy a central place. There is also ‘relative appropriateness’ to take into account, in that a given intervention may be neither appropriate nor inappropriate, in absolute terms, but may be more, or less, appropriate when examined alongside an alternative treatment.

A well-recognized definition of clinical appropriateness has been advanced by the RAND Co- operation, which is closely associated with the introduction and use of this term in the USA, some 10 years ago, in a study of the level of use of major surgical interventions. The RAND definition of clinical appropriateness is as follows:

Appropriate (care) means that the expected health benefit exceeds the expected negative consequences by a sufficiently wide margin that the procedure is worth doing.

(Brook 1991) There has, however, been criticism - both historical and ongoing - of the RAND appropriateness methodology (Davidson 1993; Park 1993; Phelps 1993; Kassirer 1994; Black 1994; Phelps 1994), and action has been taken to refine its approach, although not

without profound difficulty (Ballard 1994; Black

Some investigators have considered the RAND approach as being limited to the so-called medical perspective of health care delivery, excluding the views of patients and of society. The research led by Hop- kins, culminating in the Report to the NHS Director of Research & Development (1993), advanced the following definition:

Appropriate care means the selection, from the body of available interventions that have been shown to be efficacious for a disorder, of the intervention that is most likely to produce the outcomes desired by the individual patient. An intervention can only be appropriate when certain criteria are satisfied. The technical skills and other resources for the intervention must be available so that it can be performed to a sufficiently high standard. The intervention must be performed in a manner that is acceptable to the patient. Patients should be given adequate information about the range of effective interventions. Their preferences are central to the choice of appropriate intervention from those known to be effective. Their preferences will reflect not only the primary outcome that they hope to achieve, but also their perceptions of the potential adverse outcomes that they might encounter. It follows that patients must be fully involved in discussions about the likelihood of different outcomes with and without the intervention, and about the discomfort and other adverse effects that they might encounter. The appropriateness of healthcare interventions must also be considered within the current social and cultural context and with regard to the justice of resource allocation.

The philosophy of this definition has been elegantly demonstrated by the distinguished authors of the report using the example of coronary bypass surgery. The report argues that coronary bypass surgery is of proven efficucy in reducing mortality over 5 years in patients with left maintstream coronary disease, the best prospective studies demonstrating mortality to have been minimized to 0.5% (Chassin et al. 1986). In general operative use it is recognized that the mortality is somewhat higher, so the procedure was not as effective as had been anticipated on the basis of the

1994).

0 1996 Blackwell Science, Journal of Evaluation in Cllnical Practice, 2 , I . 13-27 21

D. O'Neill et a/.

evidence collected from various trials (Chassin et al. 1986; UC Congress and Office of Technology Assessment 1988. Larger units are able to perform operations at lower marginal cost, so they may be more efficient. For the individual patient, the operation may be quite inappropriate because the patient may wish to elect for medical treatment, or because of some serious comorbidity, or for some other reason.

The Report to the NHS Director of Research & Development (1993) was also concerned with exam- ining the dimension of clinical appropriateness in terms of the professional perspective, the lay perspective and the perspective of society as a whole.

Clinical appropriateness from the professional perspective

The professional perspective, the report advanced, will immediately be concerned with the views currently held by the health care professions on those interventions that manifestly contribute most significantly to health gain, where this is defined as the net increment in health status over the patient's life- time after subtracting health loss resulting from adverse outcomes of the intervention. The broad aspects of health care, including nursing practice and health promotion, will, the report argues, need to be considered in addition to the role of medicine itself in investigation and treatment.

Evidence base of appropriateness of procedures and interventions Appropriateness, the report advances, will need to be considered in terms of the extent to which the procedures and interventions currently employed in patient care have been examined by randomized controlled trials. Many of the procedures used, for example, within nursing care and the care administered by the professionals allied to medicine have not been subjected to the rigorous examination that has generally and traditionally been applied to medical procedures and, consequently, there is little information available about the outcomes from such care. The report recognizes that the considerable variation in rates of procedures cannot be explained by local variations in morbidity or the availability of resources alone and indicates that there is a wide variation in the working definition of appropriateness understood by health professionals in general.

Appropriateness of referrals In addition to the appropriateness of procedures administered by health care professionals within the acute setting must be considered the appropriateness of referrals to district general hospitals by general practitioners within the community setting. There is substantial evidence available within the literature demonstrating considerable variation in referral practices within general practice. The Royal College of General Practitioners (1992) found that general practitioners with a special interest refer to hospitals more patients covered by the topic of their interest than do other general practitioners, and these variations in referral characteristics persist after corrections for casemix and for demographic factors.

Measurement of clinical appropriateness from the medical perspective Appropriateness can be measured from the medical perspective. A typical example of the utility of this in practice is provided by the work of Scott & Black (1991) in relation to the appropriateness of cholecystectomy. As part of their research, these authors listed a set of 'indicators' for a particular intervention with reference to reviews of the literature. Categorization of the patient then followed with regard to symptomatology, particular history and the results of previously performed diagnostic tests, with the results of this categorization being presented to an informed panel, not all of whom were specialists. The categories were then rated for inappropriateness or appropriateness in relation to the procedure in question. Resulting discussions between panelists after their initial rating, followed by re-rating, minimized the dispersion of the ratings.

International variation in criteria for clinical appropriateness There is international variation on what is considered to be appropriate and what is not. A prominent example cited by Brook et al. (1988) described the process of rating for the appropriateness of coronary artery bypass surgery in Britain and the United States. In the United States, one panel rated coronary artery bypass surgery appropriate (median rating 7 on a 9-point scale) for a patient with angina occurring in mild exertion (class 111) receiving sub- maximal medical therapy and with a positive exercise test result, in contrast to a panel of UK physicians and cardiologists which rated the coronary bypass surgery as clearly inappropriate (median rating 2/9). The

22 0 1998 Blackwell Science. Journal of Evaluation in Clinical Practice, 2 , 1. 13-27


scientific basis for these variations is not absolutely clear, with the differences possibly reflecting not only the varying cultural values of these different societies but also the possibility that the reviews of published literature submitted to the panels did not employ proper scientific methods for generating and accessing the scientific evidence. Indeed, it is increasingly recognized that statistically unsound papers have often been afforded equivalent status to others employing far more rigorous analyses, but which may not have been made available for review.

A further perspective of importance is represented by the observations of Brook et al. (1990). These authors examined the use of endarterectomy by surgeons who frequently perform this procedure and by surgeons who do not. The research discovered that the likelihood of undergoing an appropriate endarterectomy decreased by almost one-third in patients treated by a surgeon who regularly performed this procedure in direct comparison with patients treated by a surgeon who did not routinely undertake endarterectomy. The authors additionally demonstrated that this difference derived from the former group of surgeons not only operating on acutely ill patients but also operating on less ill, symptomless patients. Of further interest is the observation by these authors that management by a gastroenterologist with a board certification, compared with another type of physician, decreased significantly the likelihood that an endoscopy would be appropriate.

Clinical appropriateness from the patient’s perspective

The patient’s perspective on appropriateness is of steadily increasing importance in modem clinical practice. Different patients will often elect to be treated according to different methods, and even when one considers a situation where two patients demonstrate a virtually identical condition, the different treatments that they choose may well be appropriate for their individual circumstances. An example that has been cited is that of the male with benign prostatic hyper- trophy. One patient with this condition may prefer to tolerate nocturnal urge and frequency rather than risk the occasional adverse outcomes of transurethral resection such as impotence and incontinence, while another may well elect for surgery. The manner in

which these positions of the patient may be incorpo- rated into clinical practice will vary.

Three methods were advanced by the Report to the NHS Director of Research and Development (1993) through which patient views may be considered in practice, and which range from the least to the most paternalistic. 1 The health professional may make a global decision, taking account of the patient’s preferences as he/ she understands them, an approach that may be summarized as ‘doctor knows best’. This method has advantages. The professional may have seen the relevant outcomes but the patient knows about them only second-hand. The patient does not need to worry about rare outcomes until they actually occur and may therefore be less anxious. He or she may actually experience a better clinical course through believing in clinical certainties rather than knowing the professional’s doubts. The disadvantages are that the health professional may be wrong about the patient’s values and, consciously or unconsciously, may sub- stitute his/her own values. When asked, patients almost always say they want to be more informed. In some circumstances, health care professionals have to act in what they believe to be their patient’s best interests - for example, when patients are unconscious. 2 The health professional may offer the patient different options and let the patient make a global decision. This method apparently respects the patient’s autonomy, but again there are difficulties. patients may make a decision considered unwise by their doctor because of poor understanding of outcomes and faulty manipulations of probabilities. On other occasions, however, particularly in chronic illness, decision making will involve patients who have developed considerable knowledge of their illness. 3 The health professional may explicitly measure the patient’s values and combine these with his/her own best estimate of the probabilities in order to choose a course of action which maximizes expected utility for that patient. This approach adopts more formal principles of decision analysis (Thornton et al. 1992).

Clinical appropriateness and the perspective of society

The advent of the NHS reforms has brought with it emphasis by the Government on the need to afford

0 1996 Blackwell Science, Journal ol Evaluation in Clinical Practice, 2. 1, 13-27 23

D. ONeill eta/.

patients an increasing right of choice in relation to the treatments that they receive within the National Health Service, rights underlined by the Patient’s Charter and the Health of the Nation. The perspective of society takes into account these events and has resulted in an increasing set of associated demands. The growing recognition by patients that clinical practices vary from clinician to clinician, and that relatively few of these will be based on solid clinical research evidence, has led to a demand for access to effective clinical care.

Hampton (1995) has pointed out that ‘patients obviously want “appropriate” treatment, and purchasers only want to pay for “appropriate” procedures, but clinical decision-making is a very individual affair, and the problem is how to bring an acceptably standard approach to patient management’.

Hampton (1995) advances the utility of decision analysis in this context, which may be described as a technique that evaluates ‘clinical uncertainty’ in quantitative terms. In a situation where a clinician is faced with two clinical options (for example, to operate or not to operate), the outcome, Hampton says, will be ‘a matter of chance, within the limits imposed by the statistical likelihood of different outcomes actually occurring’. This author provides an example focused on the use of coronary artery bypass grafting, where traditional decision making in terms of patient selection has been shown to be inconsistent (Kellett & Graham 1995). A 2% mortality might be involved, but an 80% chance of curing the patient’s symptom’s may exist. These probabilities may be compared with the predicted outcome of any alternative treatment being considered. The utility of decision analysis in routine clinical practice is thus clear in that it makes possible - and can ensure - consistency of clinical decision making. A prominent difficulty in the use of this technique is, as Hampton (1995) emphasizes, that the data required for the calculation of risks and benefits are often ‘either not available.. . out-of-date o r . . . based on inappropriate patient groups’.

In terms of data available for calculation of risks and benefits of coronary artery bypass grafting, information can be derived from three essentially small clinical trials (Takaro et al. 1976; European Coronary Surgery Study Group 1982; Alderman et al. 1990). Since the time of publication of the work by

Takaro et al. (1976) and the results of the Study Group (European Coronary Surgery Study Group 1982), the surgical procedure of coronary artery bypass grafting has changed markedly. As Hampton (1995) points out, arterial conduits are now employed in preference to venous conduits, and perfusion techniques have been substantially improved. As a consequence of these innovations, this particular surgical intervention is invested with less risk and delivers better immediate results, although it is recognized that assessment of long-term outcome will be inevitably limited in the context of a newly introduced technique. Hampton (1995) additionally emphasizes that the medical treatment of patients with coronary artery disease has changed in specta- cular fashion over the last 20 years in accordance with the implementation, albeit delayed (Haines & Jones 1994), of definitive medical evidence in terms of pro- phylactic salicylate, use of beta-blockade, use of angiotension converting enzyme inhibitors, throm- bolysis and cholesterol reduction. As a consequence, Hampton (1995) says ‘decision analysis will inevitably use incomplete and even inaccurate data for its pre- dictions, and while that might be acceptable for comparative audit, i t is not useful for making an individual decision’.

Hampton (1995) believes that, when a patient is being advised about treatment choices, the clinician is engaging in a simple form of decision analysis. In this context, this author refers to clinical judgement which involves ‘making an estimate of outcome, coupling available evidence with a guess as to how well the individual patient corresponds to the patient groups from which the evidence was derived. It is not usually too difficult to predict the average outcome in a group of patients, but the fate of an individual will inevitably depend on chance.’ Hampton concludes that ‘individual patients rely on individual decisions and while decision analysis may help base these on the best possible evidence, it is difficult for any statistical technique to allow for the all-important glint in the patient’s eye’.

Conclusion

The systematic audit of the quality of routine clinical care is now acknowledged as an integral component of good professional practice and an activity of

24 0 1996 Blackwell Science. Journal of Evaluation in Clinical Pracfice. 2 , 1, 13-27


unquestionable ethical importance. Assuming access and equity, the quality of clinical care can be critically audited in terms of its effectiveness, its appropriateness and the efficiency with which effective, appropriate care is delivered to patients. Cost-effectiveness and utilization review are as relevant in examinations of the efficiency of clinical activity as are traditional indices such as, for example, the speed of delivery of a given intervention. Of similar importance is the need to consider the patient’s view of what constitutes appropriateness in their care, an index that immediately contrasts with the traditional and paternalistic belief in the inevitable appropriateness for patients of treatments which have been shown to be objectively effective and objectively indicated for their clinical state. Efficiency and appropriateness having been considered here, we move, in the second of our two papers, to a detailed consideration of the theory and practice of the eflectiveness of clinical intervention.

Acknowledgements

Dr O’Neill thanks the members of the Utilisation Review Network for their input. Professor Miles and Dr Polychronis thank Dr Paul Bentley for helpful discussions during the course of research and writing.

References

Alderman E., Bourassa M. & Cohen L. (1990) Ten year follow-up of survival and myocardial infarction in the randomised coronary artery surgery study. Circulation 82,

Anderson P., Meara J., Broadhirst S., Attwood S., Rimbrell M., & Gatherer A. (1987) Use of beds: a cohort study of admissions to a provincial teaching hospital. British Medical Journal 297, 9 10-9 12.

Ballard D.J. (1994) The RAND/AMA/AMCC clinical appropriateness initiative: insights for multi-site appropriateness studies derived from the abdominal aortic aneurysm surgery project. International Journal of Quality in Health Care 6, 187-198.

Beech R., Challah S. & Ingram R.S. (1987) Impact of cuts in acute beds on services for inpatients. British Medical Journal 29, 685-688.

Beech R., O’Neill D.F. & Hesketh T. (1995) Analysing Acute Inpatient Services in South Thames. The Development and Application of Utilisation Review Tools. Preliminary Report. United Medical and Dental Schools Department

1-18.

of Public Health Medicine, University of London, Lon- don.

Bennet J. & Walshe K. (1990) Occurrence screening as a method of audit. British Medical Journal 300, 1248-1 25 I .

Black N. (1994) Appropriateness studies. New England Journal of Medicine 330, 432.

Bogg J, Lee A., Marie F.. Milne K., ONeill D., Seddon D., Thompson C., Thurston H. & Pearson M. (1995) Study of Emergency Medical Admissions at Aintree. University of Liverpool and North West Regional Health Authority, Liverpool.

Brook R.H. (1991) Measures of Appropriateness, Quality Care and Health Status Available from RAND. RAND, Santa Monica.

Brook R.H., Kosecoff J.B., Park R.E., Chassin M.R., Winslow C.M. & Hampton J.R. (1988) Diagnosis and treatment of coronary heart disease: doctors’ attitudes in the USA and UK. Lancet 2, 750-753.

Brook R.H., Park R.E. & Chassin, M.R. (1990) Predicting the appropriate use of carotid endarterectomy, upper gastrointestinal endoscopy and coronary angiography. New England Journal of Medicine 323, 1173-1 177.

Central Statistics Office (1994) Health Statistics. HMSO, London.

Chassin M.R., Park R.E. & Fink A. (1986) Indications for Selected Medical and Surgical Procedures - a Literature Review of Ratings and Appropriateness: Coronary Artery Bypass Graft Surgery. RAND Corporation Publication No 3204/2-CWF HF/HCFA/PMT/RWJ. RAND, Santa Monica.

Chassin M.R., Kosecoff J., Park R.E., Winslow C.M., Kahn K.L., Merrick N.J., Fink A., Keesey J., Solomon D.H. & Brook R.H. (1989) A RAND Corporation Research Study. The Appropriateness of Selected Medical and Surgical Procedures: Relationship to Geographical Variations. Health Administration Press, Ann Arbor, MI, USA.

Coast J, Inglis A., Morgan K, Gray S., Kammerling M. & Frankel S . (1994) The Hospital Admissions Study: Final Report. University of Bristol, Bristof.

Coid J. & Crome P. (1986) Bed blocking in Bromley British Medical Journal 292, 2 153-2 1 56.

Crombie D.L. & Cross K.W. (1959) Implicit criteria instruments. Medical Practice 242, 361-364.

Davidson G. (1993). Does inappropriate use explain small area variations in the use of healthcare services? A critique. Health Services Research 28, 401405.

Demlo L.K. (1983) Assuring quality of health care. An overview. Evaluation and the Health Professions 6 , 161- 196.

Department of Health (1994) Developing NHS Purchasing and GP Fundholding: Towardr a Primary Care Led NHS. Department of Health EL(94)79, London.

Q 1996 Blackwell Science, Journal of Evaluafron in Clinical Practice. 2, 1, 13-27 25

D. ONeill et a/.

Dippe S.E., Bell M.M. & Wells M. (1989) A peer-review of peer review organisations. West Journal Medicine 51, 561.

Donabedian A. (1982) Explorations in Quality Assessment and Monitoring. The Criteria and Stanabrds of Quality (II) Health Administration Press, Ann Abor, MI, USA.

Donaldson S.N., Wheeler M.R. & Barr A. (1972) Demand for patient care. British Medical Journal 212, 799-802.

Ehreth J. (1994) The development and evaluation of hospital performance measures for policy analysis. Medical Care 32, 568-597.

European Coronary Surgery Study Group (1982) Long term results of a prospective randomised study of coronary bypass surgery in stable angina pectoris. Lancet 2, 1173- 1180.

Farag R.R. & Tinker G.M. (1985) Delays in discharge of patients from an acute geriatric unit. Health Trends 17, 44.

Fenn P. & Mayhew K. (eds) (1991) Providing Healthcare. The Economies of Alternative Systems of Finance and Delivery. Oxford University Press, Oxford.

Fenn A., Horner P., Travis S., Prescott G. & Bates T. (1995) Inuppropriate Bed Usage in a District General Hospital. Preliminary Report. William Harvey Hospital and East Kent Health Authority, Kent.

Forsyth G. & Logan, R.F.L. (1960) The Demand for Med- ical Care. Churchill Livingstone, London.

Gertman P.M. & Restuccia J.D. (1981) The appropriateness evaluation protocol: a technique for assessing unneces- sary days of hospital care. Medical Care 8, 855-871.

Haines A. & Jones R. (1994) Implementing findings of research. British Medical Journal 308, 1488-1492.

Hampton J.R. (1989) Inappropriate Investigation and Treatment in Clinical Practice. Conference Report. Royal College of Physicians, London.

Hampton J.R. (1995) Practice variations, appropriateness and decision analysis. Quarterly Journal of Medicine 88,

Harvey J., Jenkins R. & Llewelyn L. (1993) Estimating the appropriateness of acute bed use: role of the patient hotel. Journal of Epidemiology and Community Health 41, 368- 372.

Hobbs R. (1995) Rising emergency admission. British Medical Journal 310, 207-208.

Hopkins A. (1989) Appropriate investigation and treatment in clinical practice. Royal College of Physicians, London.

Houghton A. (1994) The audit of medical admissions. Royal College of Physicians, London.

Jarvis E. (1850) The influence of distance from and proxi- mity to an insurance hospital on its use by any people. Boston Medicine Surgery Journal 42, 226-232.

Kassirer J.P. (1994) Appropriate studies. New England Journal of Medicine 329, 1263-1267.

365-367.

Kellet J. & Graham J. (1995) Audit,of a single clinician’s referral decisions for coronary angiography after exercise testing: comparison of traditional decision-making with a decision-analysis model. Quarterly Journal of Medicine 88,401408.

Kempner K.J. (1 988) Medically inappropriate hospital use in a paediatric population. New England Journal of Medicine 321, 1033-1037.

Krakauer H., Bailey Clifton R., Cooper H., Ms, Wai-Kouk, Yu, Skellan, K.J. & Kattakkuzhy G. (1995) The systematic assessment of variations in medical practices and their outcomes. Public Health Reports 110, 2-12.

Lawrence D. & Edwards G. (1995) Emergency Admissions and Bed Use. Buckinghamshire Health Authority, Buck- inghamshire.

Liberati, A. (1995) Appropriateness of Hospital Use: an EC Biomed European Project Assessing the Medical Appro- priateness of Hospital Utilisation and its Structural, Financial and Social Determinants. Progress Report. Mario Negri Institute, Milan.

Liberati A. & Members of the Steering Group and Coor- dinating Centre for Appropriateness of Hospital Use and the European Biomed Project (1995) A series of 12 articles from the European Collaborative Group. International Journal of Quality in Health Care I, 185-276.

Mackintosh J . M . & Garrat F.N. (1961) An examination of the need for hospital admission. Lancet 2, 8 15-8 18.

Miles A., Lugon M. & Polychronis A. (1996a). Clinical audit and the development of quality in clinical practice - I . In Effective Clinical Prnctice (eds A. Miles & M. Lugon), in press. Blackwell Science Ltd, Oxford.

Miles A., Lugon M. & Polychronis A. (1996b) Clinical audit and the development of quality in clinical practice - 11. In Effective Clinical Practice (eds A. Miles & M. Lugon), in press. Blackwell Science Ltd, Oxford.

Miles A., Bentley D.P., Price N., Polychronis A., Grey J. & Asbridge J. (1996~) The Total Healthcare Audit System: a systematic methodology for clinical practice evaluation and development in NHS provider organizations. Journal of Evaluation in Clinical Practice 2, 37-64.

Miles A,, Bentley D.P., Price N., Polychronis A., Grey J.E. & Asbridge J. (l996d) Clinical practice evaluation in the NHS: methodologies past and present - I. In Effective Clinical Practice (eds A. Miles & M. Lugon), in press. Blackwell Science Ltd, Oxford.

Miles A., Bentley D.P., Price N., Polychronis A., Grey J.E. & Asbridge J. (l996e). Clinical practice evaluation in the NHS: methodologies past and Present - 11. In Eflective Clinical Practice (eds A. Miles & M. Lugon), in press. Blackwell Science Ltd, Oxford.

Miles A., O’Neill D. & Polychronis A. (19960 Central dimensions of clinical practice evaluation: efficiency.

26 0 1996 Blackwell Science, Journal of Evaluation in Clinical Practice. 2 , 1 , 13-27


appropriateness and effectiveness-11. Journal of Eva- luation in Clinical Practice 2, in press.

Murphy R.W. (1977) Blocked beds. British Medical Journal 263, 1395-1396.

Murphy M. & Charlton J. (1995) Health of Adult Britain: Trenh in all Causes of Morfality. Office of Population Census and Survey, in press.

National Health Service Management Executive (1991) Executive Letter EL(92)91. HMSO, London.

Park R.E. (1993) Does inappropriate use explain small area variations in the use of healthcare services? Health Service Research 28, 401.

Payne S.M. (1987) Identifying and managing inappropriate hospital utilisation. A policy synthesis. Health Services Research 22, 5.

Pearson R.J.C., Smedby B., Befernstein R., Logan R.F.L., Burgess A.M. & Peterson O.L. (1968) Hospital caseloads in Liverpool, New England and Upopsala; an inter- nationalised comparison. Lancet 2, 559-566.

Phelps C.E. (1993) The methodological foundation of studies of the appropriateness of medical care. New England Journal of Medicine 330, 433.

Phelps C.E. (1994) Appropriateness studies. New England Journal of Medicine 330, 433.

Relman AS. ( 1 988) Assessment and accountability: the third revolution. New England Journal of Medicine 319, 122& 1222.

Report to the NHS Director of Research and Development (1993). HMSO, London.

Restuccia J.D. (1982) The effect of concurrent feedback in reducing inappropriate hospital utilisation. Medical Care 1, 46-62.

Royal College of General Practitioners (1992) European Study of Referrals from Primary to Srcondary Care. Report to the Concerted Action Committee of Health Services Research for the European Community. Occa- sional Paper 56. Royal College of General Practitioners, London.

Rosser R.M. (1976) The reliability and application of clinical judgement in evaluating the use of hospital beds. Medical Care XIVI.

Sanazaro P.J. (1980) Quality assessment and quality assurance in medical care. Annual Review of Public Health 1,

Scott E.A. & Black, N.A. (1991) Appropriateness of cholecystectomy - a consensus panel approach. Gut 32, 1066- 1070.

Takaro T., Hultgren H.N. & Lipton M.J. (1976) The VA cooperative randomised study of surgery for coronary arterial occlusive disease. Circulation 54 (Suppl. 3), 107- 117.

Thornton, J. Lilford R. & Johnson N. (1992). Decision analysis in medicine. British Medical Journal 304, 1099- 1103.

Torrance N., Hogg D. & Knox J.D.E. (1972) Acute admissions to medical beds. Journal of the Royal College of General Practitioners 22, 2 1 1-2 19.

Utilisation Management Associates (1993) Managed Care Appropriateness Protocol. U.M.A., USA.

US Congress and Office of Technical Assessment (1988) The Quality of Medical Care: Information for Consumers. US Government Printing Ofiice, Washington. Publication

US Department of Health, Education and Welfare (1974) Office of Professional Standards Review: PRSO Program Manual. Government Printing Office, Washington.

Victor C. & Khanoo A.A. (1994) Is hospital the right place? A survey of inappropriate admissions to an inner London NHS Trust. Journal of Public Health Medicine 16, 2 8 6 290.

Victor C., Nazareth B. & Hudson M. (1993) The inappropriate use of acute beds in an inner London district health authority. Health Trends 25, 94-97.

Wennberg J.E. & Gittelsohn A. (1982) Variations in medical care among small areas. Scientijic American 246, 12&135.

Yi J., Rowley M.J., Hampton J.R. (1989) Appropriate use of investigations in cardiology. In Appropriate Investiga- tions and Treatment in Clinical Practice (ed. A. Hopkins), pp. 31-39. Royal College of Physicians, London.

37-38.

NO OTA-H-386.

fo 1996 Blackwell Science, Journal of Evaluation /n Clmcal Practtce. 2 , 1, 13-27 27

Documents

Central dimensions of clinical practice evaluation: efficiency, appropriateness and effectiveness - I