Copyright ( 1999 John Wiley & Sons, Ltd.

STATISTICS IN MEDICINE

Statist. Med. 18, 487—490 (1999)

STATISTICS IN THE MEDICAL LITERATURE: 3

Editor: Douglas Altman

RATIONALE AND CALL FORCONTRIBUTIONS

After two appearances1,2 this section of Statisticsin Medicine has been in temporary hibernation.Because of the long gap since the last such compi-lation, I am including here an updated account ofthe aims of the section.

Papers of potential interest to medical statisti-cians can appear in any medical journal. Examplesinclude reviews of the (mis)use of statistics, didacticarticles, editorials on the use of statistical methods,papers considering ethical aspects of research, andsometimes even new methodology. Also, somemedical research papers may be interestingfor teaching purposes, because of good, or, morelikely, bad features of design analysis or inter-pretation. The intention of this section in Statisticsin Medicine is to draw attention to some of thesepapers, with no ambition to be comprehensive. Ingeneral the emphasis will be on papers in clinicaljournals (broadly interpreted) rather than epi-demiology journals. The focus will be on recentpublications, but here too there is likely to besome latitude.

Suggestions for papers to highlight would bemost welcome, particularly if published in moreobscure journals and especially if accompanied bya brief explanation of the features of interest if thisis not obvious. Even better would be fully preparedcontributions — I certainly do not wish to remainthe only contributor. It would be helpful if anyoneinterested in contributing could let me know whichpapers they wish to write about, to avoid possibleduplication of effort.

GUIDELINES FOR REPORTING STUDIES

There have been many publications offering stat-istical advice relating to particular types of studydesign or analysis. More recently, attention has

focused on guidelines for reporting studies andcritical appraisal of published studies. The distinc-tion between these types of guidelines is rathervague however. As anyone who has tried to devel-op guidelines will know, it is at best difficult todivorce advice on reporting research from com-ments on how to do (and how not to do) research.

An important recent example is the CONSORTstatement3 for reporting randomized trials. This isunique in having been ‘adopted’ by over 70 jour-nals by the end of 1997.4 Here adoption impliesthat journals require authors to comply with therecommendations for reporting trials. This statusmay at least partly reflect the fact that, unusually,journal editors were among the authors, but itprobably also relates to the widespread recogni-tion that the reporting of controlled trials is gener-ally inadequate for those carrying our systematicreviews. The beneficial effects of CONSORT arealready being in seen in many journals. It is clear,though, that they will need to be modified in thelight of experience.4,5

A recent paper in the Japanese Journal of Clini-cal Oncology presents general guidelines coveringmany aspects of medical research.6 The two sec-tions are headed ‘Basic requirements for studyreports’ and ‘Common pitfalls in statistics’. Thelatter, which is twice as long as the first section,covers statistical significance, association andcausal relationship, distribution and underlyingassumptions, categorical variables, correlation andregression, and survival analysis. The authors offermuch sensible advice, some drawn from earlierpublications. They are unusual in including illus-trative graphs and also in the fact that they are alsoavailable on the Internet (in both English andJapanese).6

It is worth mentioning here too the valuablebook by Lang and Secic,7 which almost certainlyprovides the most comprehensive current guide-lines for statistical reporting.

QUALITY OF PUBLISHED STUDIES

The need for CONSORT is well illustrated bya review of the quality of 122 randomized trialscomparing selective serotonin reuptake inhibitors(SSRIs) and tricyclic or heterocyclic antidepress-ants.8 As the authors note, the use of SSRIs asa first line treatment of depression remains un-resolved despite several published meta-analyses.The findings of their systematic review of themethodology of these trials makes dismal reading.Of the 122 trials, just one had an adequate descrip-tion of the randomization procedure, only 11 percent of trials had adequate power, only four trialshad a follow-up period of more than eight weeks,69 per cent of studies made multiple comparisons(usually of multiple outcome measures) and abouthalf the studies failed to carry out a proper inten-tion-to-treat analysis. In addition, only one paperincluded an economic analysis and only twolooked at quality of life.

ORDINAL LOGISTIC REGRESSION

Logistic regression is now used very widely inmedical research (see below) but there seem to besurprisingly few published articles introducing themethods for a medical readership. One such waspublished a few years ago in the Journal of theRoyal College of Physicians of ¸ondon,9 and nowthe same journal has published a paper on ordinallogistic regression. Bender and Grouven10 give‘a non-technical introduction to the propor-tional odds model’. While this is a valid claim,I suspect that medical readers may struggle a bitwith some of the mathematics. However, this paperis likely to interest those already at the more nu-merate end of the medical spectrum. The authorsapproach ordinal logistic regression via ordinary(binary) logistic regression and polytomous logis-tic regression, thus placing the problem and thesolution in appropriate context. They note theabsence of a global goodness-of-fit test in standardstatistical software and the strong assumptionsunderlying the proportional odds model, and sug-gest that modelling should begin with examinationof separate binary logistic models. There is a clearworked example, and the paper ends with recom-mendations for the presentation of results of suchanalyses in published papers.

CHANGES IN THE USE OF STATISTICS

Although there have been numerous published re-views of the use of statistics in medical journals,

few have had a longitudinal design. Two suchstudies have recently been published.

Seldrup11 reported a comparison of the use ofstatistics in the British Medical Journal in 1977 and1994. Not surprisingly there had been major chan-ges over the 17 years. Some of the more markedincreases were in papers with statistical authors (10per cent to 25 per cent), papers citing statisticalliterature (9 per cent to 47 per cent), reference tosoftware (1 per cent to 36 per cent), reporting ofconfidence intervals (4 per cent to 62 per cent), anduse of logistic regression (1 per cent to 16 per cent).Inclusion of orphan P-values, without identifica-tion of the test used, had fallen from 30 per cent to6 per cent.

While the general picture was one of consider-able improvement, Seldrup closed by noting themany recent methodological advances, but ob-served that ‘2confidence in the results of medicalresearch could be undermined through undue reli-ance on statistical manipulations (adjustments) tocorrect for less well planned, less well designed,and less well thought out research’.

Another longitudinal study, by Williams andcolleagues,12 compared the use of statistics in theAmerican Journal of Physiology: Heart and Circula-tory Physiology in 1984 and 1994. Whereas Sel-drup explicitly avoided making judgements aboutcorrect usage of statistics, Williams and colleaguestackled this question head on, in particular withrespect to the use of t-tests and the power of tests.Unlike Seldrup, they found more similarities thandifferences between the two years. They found thatin both years about half of the papers examinedincluded at least one t-test with a power of lessthan 60 per cent to detect a 20 per cent differencebetween group means. Multiple t-tests were usedrather than analysis of variance in 18 per cent and16 per cent of papers. They commented on thecommon failure of authors to identify whichmethod of analysis was used where, and generalinadequate description of experimental design.

SCIENCE AND ETHICS

A few years ago Rosenthal13 explored the relationbetween scientific quality and ethics in research.His focus was psychological research but his obser-vations apply generally. Among the more obviousissues that he discussed are: the need for ethicscommittees (institutional review boards) to takemore note of scientific issues; the dangers of drop-ping observations and multiple testing; the misrep-resentation of findings; and the failure to report

488 STATISTICS IN THE MEDICAL LITERATURE: 3

Copyright ( 1999 John Wiley & Sons, Ltd. Statist. Med. 18, 487—490 (1999)

data that are not in line with expectation. As henoted, ‘Failing to report data that contradict one’searlier research, or one’s theory or one’s values, ispoor science and poor ethics.’ He also made someinteresting comments on meta-analysis: that itretroactively increases the value of the (primary)studies being summarized, and that meta-analysisis an ethical imperative. A more unusual argumentrelated to applying cost-utility analysis to deter-mine which research should be done. Having pre-viously proposed such an approach here he arguesthat it omits a key issue, ‘the costs (and utilities) ofnot conducting a particular study’. He observesthat ‘the failure to conduct a study that could beconducted is as much an act to be evaluated onethical grounds as is conducting a good study’.This argument seems to relate to the mismatchbetween the burden of disease and the focus ofresearch activity.

I have often wondered how many of the minorscientific misdemeanours that statisticians fre-quently encounter stem from researchers havinga different view from statisticians of what consti-tute acceptable practices. A study by Korenman etal.14 sheds light on this area, and picks up many ofthe themes that Rosenthal discussed. They sur-veyed 924 scientists and 140 institutional represen-tatives (officials from the scientists’ institutions),with a 69 per cent response rate. They used a frac-tional factorial design, in which each respondentreceive a random sample of 12 from a 8364 pos-sible scenarios, generated using random selectionof phrases. They examined four areas: performanceand reporting of research; appropriation ofideas; conflict of interest, and collegiality andsharing. As well as asking respondents to judgewhether each scenario was (un)ethical, theywere asked to grade behaviour on a ‘malfeasance’scale from 1 (barely unethical) to 10 (extremelyunethical). Fabrication and falsification of datawere almost universally condemned, yet even herea few individuals did not consider such behaviourunethical. Among the scientists, 1)4 per centdid not deem unethical the provision of a mislead-ing explanation of how the study was done tomake it look sounder than it was; 12 per cent didnot deem unethical incomplete reporting of re-search which made it impossible to replicate thework in other laboratories, and 34 per cent did notdeem unethical picking the best results to reportbecause he/she honestly believes them to be thecorrect ones. The institutional representatives hadsimilar views about ethics, but were much morelikely to believe that unethical behaviour deservedpunishment.

One issue mentioned more than once is thedesirability of educating researchers in the ethics ofresearch. This study also raises the fascinatingquestion about whether being ethical is a binary orcontinuous attribute.

AVAILABILITY OF RESEARCH DATA

Statisticians often wish to have access to the datafrom published studies. Most obviously this hap-pens when carrying out a systematic review,for which inadequate reporting of the primarystudies seriously hampers attempts to combinedata from several studies, but access to data maybe desirable for other reasons, including reanalysisusing some alternative statistical method or forteaching purposes. As many of us have found, dataare often either not available (lost) or notmade available, perhaps for logistic or commercialreasons.

Kirwan15 contacted 25 pharmaceutical com-panies with an interest in rheumatology to ascer-tain their attitudes to making data available. Just5 of the 21 who replied agreed with the idea that‘authors of a published clinical study deposit thedata relating to that publication in a data bankaccessible to their colleagues’ (that is, to otherresearchers). Over half of the respondents felt thatsuch a scheme would lead to difficulties associatedwith commercially sensitive information, althoughKirwan felt that some at least may have failed torealize that the proposal related only to data usedin a particular publication.

The issue of data availability of course goesmuch wider than the pharmaceutical industry, andhas also been aired in recent years in the BritishMedical Journal.16,17

Douglas G. Altman

REFERENCES

1. Altman, D. G. ‘Statistics in the medical litera-ture: 1’, Statistics in Medicine, 14, 873—874(1995).

2. Altman, D. G. ‘Statistics in the medical litera-ture: 2’, Statistics in Medicine, 15, 559 (1996).

3. Begg, C., Cho, M., Eastwood, S., Horton, R.,Moher, D., Olkin, I., Pitkin, R., Rennie, D.,Schulz, K. F., Simel, D. and Stroup, D. F.‘Improving the quality of reporting of ran-domized controlled trials: the CONSORTStatement’, Journal of the American MedicalAssociation, 276, 637—639 (1996).

STATISTICS IN THE MEDICAL LITERATURE: 3 489

Copyright ( 1999 John Wiley & Sons, Ltd. Statist. Med. 18, 487—490 (1999)

4. Moher, D. ‘CONSORT: an evolving tool tohelp improve the quality of reports of random-ized controlled trials’, Journal of the AmericanMedical Association, 279, 1489—1491 (1998).

5. Meinert, C. ‘Beyond CONSORT: Need forimproved reporting standards for clinicaltrials’, Journal of the American Medical Associ-ation, 279, 1487—1489 (1998).

6. Fukuda, H. and Ohashi, Y. ‘A guideline forreporting results of statistical analysis in Ja-panese Journal of Clinical Oncology’, JapaneseJournal of Clinical Oncology, 27, 121—127(1997). [Also available at http://wwwinfo.ncc.go.jp/jjco/].

7. Lang, T. A. and Secic, M. How to Report Stat-istics in Medicine. Annotated Guidelines forAuthors, Editors, and Reviewers, AmericanCollege of Physicians, Philadelphia, 1997.

8. Hotopf, M., Lewis, G. and Normand, C.‘Putting trials on trial — the costs and conse-quences of small trials in depression: a system-atic review of methodology’, Journal ofEpidemiology and Community Health, 51,354—358 (1997).

9. Hall, G. H. and Round, A. P. ‘Logistic regres-sion—explanation and use’, Journal of theRoyal College of Physicians of ¸ondon, 28,242—246 (1994).

10. Bender, R. and Grouven, U. ‘Ordinal logisticregression in medical research’, Journal of theRoyal College of Physicians of ¸ondon, 31,546—551 (1997).

11. Seldrup, J. ‘Whatever happened to the t-test?’,Drug Information Journal, 31, 745—750 (1997).

12. Williams, J. L., Hathaway, C. A., Kloster, K. L.and Layne, B. H. ‘Low power, type II errors,and other statistical problems in recent cardio-vascular research’, American Journal of Physi-ology, 273, H487—H493 (1997).

13. Rosenthal, R. ‘Science and ethics in conduct-ing, analyzing, and reporting psychological re-search’, Psychological Science, 5, 127—134(1994).

14. Korenman, S. G., Berk, R., Wenger, N. S. andLew, V. ‘Evaluation of the research norms ofscientists and administrators responsible foracademic research integrity’, Journal of theAmerican Medical Association, 279, 41—47 (1998).

15. Kirwan, J. R. ‘Making original data from clini-cal studies available for alternative analysis’,Journal of Rheumatology, 24, 822—825 (1997).

16. Davey Smith, G. ‘Increasing the accessibilityof data’, British Medical Journal, 308,1519—1520 (1996).

17. Delamothe, T. ‘Whose data are they anyway?’,British Medical Journal, 312, 1241—1242 (1996).

490 STATISTICS IN THE MEDICAL LITERATURE: 3

Copyright ( 1999 John Wiley & Sons, Ltd. Statist. Med. 18, 487—490 (1999)