The reporting of diagnostic accuracy in obstetrics and gynecology literature an overview on the report quality between 1980 and 2005 : Class1 :Ana Costa,

The reporting of diagnostic accuracy in obstetrics and gynecology literature

an overview on the report quality between 1980 and 2005

Class1 : :Ana Costa, André Sena, Duarte Pinto, Inês Brandão, João Silva, Lídia Sousa, Luís Barros, Márcia Marinho, Maria Rocha, Pedro Falcão, Rafael Silva, Sara Tapa, Susana Franco.

Supervisors: Altamiro da Costa Pereira, MD, PhD; Cristina Santos, MD, PhD

Introduction to Medicine; Porto Faculty of Medicine 2006/2007

Objectives Methods AcknowledgmentsResultsIntroduction

The reporting of diagnostic accuracy in obstetrics and gynecology literature: an overview on the report quality between 1980 and 2005 2006/2007

IntroductionIntroduction

Obstetrics and Gynecology are branches of medicine(Cunningham, 2005 and Berek, 1996)

These areas depend on diagnostic tests(Bossuyt, 2003)

Studies of new diagnostic tests, and their improvements, are reported in scientific articles

This studies, and their reporting, are not always well design and this may lead to unwanted consequences(Bossuyt, 2003)

Objectives Methods AcknowledgmentsResults

Cunningham, F.Gary. Williams obstetrics, 26 edição (2005)

Berek, Jonathan S. Novak’s gynecology 12 edição (1996)

Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.Annuals of internal Medicine. 2003


IntroductionIntroduction

Studies to determinate the diagnostic accuracy are a vital part in the rigorous evaluation process of diagnostic tests(Bossuyt, 2003 and Deeks, 2001)

The diagnostic accuracy can be expressed in many ways, including sensitivity and specificity, likelihood ratios and diagnostic odds ratio(Bossuyt, 2003 and Deeks, 2001)

STARD was created to guideline the reporting of studies of diagnostic accuracy(Smidt, 2006)

Objectives Methods AcknowledgmentsResults

Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003

Deeks J. Systematic reviews in health care: systematic reviews of diagnostic tests. BMJ 2001

Smidt N, Rutjes A WS, et al. Reproducibility of the STARD checklist: an instrument to assess the quality of reporting of diagnostic accuracy studies. BMC 2006


To analyze the evolution of the reporting of diagnostic accuracy in obstetrics and gynecology literature (according to STARD parameters), between 1980 and 2005

To check the improvements that have being made and what still needs to be improved.

Introduction ObjectivesObjectives Methods AcknowledgmentsResults


Articles’ search

Articles’ selection

Articles’ evaluation

Statistical analyze

Introduction Objectives MethodsMethods AcknowledgmentsResults


Articles’ searchArticles’ search



Statistical analyze


(((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR

"screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR

"predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference

values"[All Fields]) OR "reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR

"roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR "likelihood

ratio"[All Fields])

Standard query to find articles that report studies of diagnostic accuracy (in PubMed)

Devillé, W. L. et. al., Conducting systematic reviews of diagnostic studies: didactic guidelines. BMC Medical Research Methodology 2002



Search for the journals

query: gynecology[Subject Terms] OR obstetrics[Subject Terms]





Statistical analyze


(((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR "likelihood ratio"[All Fields]) AND ("Acta Ginecol (Madr)"[Journal] OR "Acta Gynaecol Obstet Hisp Lusit"[Journal] OR "Acta Obstet Ginecol Hisp Lusit"[Journal] OR "Acta Obstet Ginecol Hisp Lusit Suppl"[Journal] OR "Acta Obstet Gynaecol Jpn"[Journal] OR "Acta Obstet Gynecol Scand"[Journal] OR "Acta Obstet Gynecol Scand Suppl"[Journal] OR "Akush Ginekol (Mosk)"[Journal] OR "Akush Ginekol (Sofiia)"[Journal] OR "Am J Obstet Gynecol"[Journal] OR "An Bras Ginecol"[Journal] OR "Ann Chir Gynaecol"[Journal] OR "Ann Chir Gynaecol Fenn"[Journal] OR "Ann Chir Gynaecol Fenn Suppl"[Journal] OR "Ann Chir Gynaecol Suppl"[Journal] OR "Ann Ostet Ginecol Med Perinat"[Journal] OR "Arch Gynakol"[Journal] OR "Arch Gynecol"[Journal] OR "Arch Gynecol Obstet"[Journal] OR "Arch Ostet Ginecol"[Journal] OR "Asia Oceania J Obstet Gynaecol"[Journal] OR "Attual Ostet Ginecol"[Journal] OR "Aust Coll Midwives Inc J"[Journal] OR "Aust J Midwifery"[Journal] OR "Aust N Z J Obstet Gynaecol"[Journal] OR "Aust N Z J Obstet Gynaecol Suppl"[Journal] OR "AWHONN Lifelines"[Journal] OR "AWHONN Voice"[Journal] OR "AWHONNS Clin Issues Perinat Womens Health Nurs"[Journal] OR "Baillieres Best Pract Res Clin Obstet Gynaecol"[Journal] OR "Baillieres Clin Obstet Gynaecol"[Journal] OR "Best Pract Res Clin Obstet Gynaecol"[Journal] OR "Bibl Gynaecol"[Journal] OR "Biol Res Pregnancy Perinatol"[Journal] OR "Birth"[Journal] OR "Birth Gaz"[Journal] OR "BJOG"[Journal] OR "Br J Obstet Gynaecol"[Journal] OR "Breast Dis"[Journal] OR "Bull Am Coll Nurse Midwifery"[Journal] OR "Bull Am Coll Nurse Midwives"[Journal] OR "Bull Fed Soc Gynecol Obstet Lang Fr"[Journal] OR "Bull Soc R Belge Gynecol Obstet"[Journal] OR "Cesk Gynekol"[Journal] OR "Ceska Gynekol"[Journal] OR "Climacteric"[Journal] OR "Clin Exp Hypertens B"[Journal] OR "Clin Exp Obstet Gynecol"[Journal] OR "Clin Ginecol"[Journal] OR "Clin Obstet Gynaecol"[Journal] OR "Clin Obstet Gynecol"[Journal] OR "Complement Ther Clin Pract"[Journal] OR "Complement Ther Nurs Midwifery"[Journal] OR "Contracept Fertil Sex"[Journal] OR "Contrib Gynecol Obstet"[Journal] OR "Curr Opin Obstet Gynecol"[Journal] OR "Curr Pract Obstet Gynecol Nurs"[Journal] OR "Diagn Gynecol Obstet"[Journal] OR "Early Pregnancy"[Journal] OR "Early Pregnancy"[Journal] OR "Eur J Gynaecol Oncol"[Journal] OR "Eur J Obstet Gynecol Reprod Biol"[Journal] OR "Fetal Diagn Ther"[Journal] OR "Fetal Ther"[Journal] OR "Fortschr Geburtshilfe Gynakol"[Journal] OR "Geburtshilfe Frauenheilkd"[Journal] OR "Ginecol Obstet Mex"[Journal] OR "Ginekol Pol"[Journal] OR "Gynaecologia"[Journal] OR "Gynakol Geburtshilfliche Rundsch"[Journal] OR "Gynakol Rundsch"[Journal] OR "Gynakologe"[Journal] OR "Gynecol Endocrinol"[Journal] OR "Gynecol Invest"[Journal] OR "Gynecol Obstet (Paris)"[Journal] OR "Gynecol Obstet Fertil"[Journal] OR "Gynecol Obstet Invest"[Journal] OR "Gynecol Oncol"[Journal] OR "Gynecol Prat"[Journal] OR "Gynecologie"[Journal] OR "Hypertens Pregnancy"[Journal] OR "Infect Dis Obstet Gynecol"[Journal] OR "Int J Biol Res Pregnancy"[Journal] OR "Int J Fertil"[Journal] OR "Int J Fertil Menopausal Stud"[Journal] OR "Int J Fertil Womens Med"[Journal] OR "Int J Gynaecol Obstet"[Journal] OR "Int J Gynecol Cancer"[Journal] OR "Int J Gynecol Pathol"[Journal] OR "Int J Obstet Anesth"[Journal] OR "Int Urogynecol J Pelvic Floor Dysfunct"[Journal] OR "J Am Assoc Gynecol Laparosc"[Journal] OR "J Am Coll Surg"[Journal] OR "J Br Menopause Soc"[Journal] OR "J Gynecol Obstet Biol Reprod (Paris)"[Journal] OR "J Gynecol Surg"[Journal] OR "J Hum Lact"[Journal] OR "J Jpn Obstet Gynecol Soc"[Journal] OR "J Low Genit Tract Dis"[Journal] OR "J Matern Fetal Med"[Journal] OR "J Matern Fetal Neonatal Med"[Journal] OR "J Midwifery Womens Health"[Journal] OR "J Minim Invasive Gynecol"[Journal] OR "J Obstet Gynaecol"[Journal] OR "J Obstet Gynaecol"[Journal] OR "J Obstet Gynaecol Br Commonw"[Journal] OR "J Obstet Gynaecol Can"[Journal] OR "J Obstet Gynaecol India"[Journal] OR "J Obstet Gynaecol Res"[Journal] OR "J Obstet Gynecol Neonatal Nurs"[Journal] OR "J Pediatr Adolesc Gynecol"[Journal] OR "J Psychosom Obstet Gynaecol"[Journal] OR "J Sex Med"[Journal] OR "J Soc Gynecol Investig"[Journal] OR "J SOGC"[Journal] OR "J Womens Health"[Journal] OR "J Womens Health (Larchmt)"[Journal] OR "J Womens Health Gend Based Med"[Journal] OR "Josanpu Zasshi"[Journal] OR "Jugosl Ginekol Opstet"[Journal] OR "Jugosl Ginekol Perinatol"[Journal] OR "Major Probl Obstet Gynecol"[Journal] OR "Med Gynaecol Androl Sociol"[Journal] OR "Menopause"[Journal] OR "Midwife Health Visit"[Journal] OR "Midwifery"[Journal] OR "Midwifery Today Childbirth Educ"[Journal] OR "Midwifery Today Int Midwife"[Journal] OR "Midwives"[Journal] OR "Midwives Chron"[Journal] OR "Minerva Ginecol"[Journal] OR "Mod Midwife"[Journal] OR "NAACOG Tech Bull"[Journal] OR "Ned Tijdschr Verloskd Gynaecol"[Journal] OR "Nippon Sanka Fujinka Gakkai Zasshi"[Journal] OR "Nurs Mirror Midwives J"[Journal] OR "Obstet Ginecol (Bucur)"[Journal] OR "Obstet Ginecol Lat Am"[Journal] OR "Obstet Gynecol"[Journal] OR "Obstet Gynecol Annu"[Journal] OR "Obstet Gynecol Clin North Am"[Journal] OR "Obstet Gynecol Surv"[Journal] OR "Pak J Surg Gynaecol Obstet"[Journal] OR "Patol Clin Ostet Ginecol"[Journal] OR "Pediatr Akus Ginekol"[Journal] OR "Placenta"[Journal] OR "Placenta Suppl"[Journal] OR "Pract Midwife"[Journal] OR "Prenat Diagn"[Journal] OR "Prim Care Update Ob Gyns"[Journal] OR "Prof Care Mother Child"[Journal] OR "Quad Clin Ostet Ginecol"[Journal] OR "RCM Midwives"[Journal] OR "RCM Midwives J"[Journal] OR "Ref Gynecol Obstet"[Journal] OR "Rev Chil Obstet Ginecol"[Journal] OR "Rev Clin Inst Matern Lisb"[Journal] OR "Rev Colomb Obstet Ginecol"[Journal] OR "Rev Esp Obstet Ginecol"[Journal] OR "Rev Fr Gynecol Obstet"[Journal] OR "Rev Ginecol Obstet (Sao Paulo)"[Journal] OR "Rev Mex Cir Ginecol Cancer"[Journal] OR "Rev Obstet Ginecol Venez"[Journal] OR "Rev Pediatr Obstet Ginecol Obstet Ginecol"[Journal] OR "Riv Ital Ginecol"[Journal] OR "Riv Ostet Ginecol"[Journal] OR "Riv Ostet Ginecol Prat"[Journal] OR "Riv Ostet Ginecol Prat Med Perinat"[Journal] OR "Rozhl Chir"[Journal] OR "Sanfujinka No Jissai"[Journal] OR "Soins Gynecol Obstet Pueric"[Journal] OR "Soins Gynecol Obstet Pueric Pediatr"[Journal] OR "Suppl Int J Gynecol Obstet"[Journal] OR "Surg Gynecol Obstet"[Journal] OR "Taiwan J Obstet Gynecol"[Journal] OR "TeLindes Oper Gynecol Updates"[Journal] OR "Trans N Engl Obstet Gynecol Soc"[Journal] OR "Trans Pac Coast Obstet Gynecol Soc"[Journal] OR "Ultrasound Obstet Gynecol"[Journal] OR "Vopr Okhr Materin Det"[Journal] OR "Women Birth"[Journal] OR "Women Health"[Journal] OR "Womens Health Data Book"[Journal] OR "Womens Health Issues"[Journal] OR "Womens Health J"[Journal] OR "Z Geburtshilfe Perinatol"[Journal] OR "Zentralbl Gynakol"[Journal] OR "Zhonghua Fu Chan Ke Za Zhi"[Journal])

Final query (after addition of terms to specify the search to the journals)





Statistical analyze


Limits to the search

Periods (one for each selector):- 2005 to 2005- 2004 to 2004- 2000 to 2003- 1990 to 1999- 1980 to 1989

Language restriction- English- Portuguese





Statistical analyze

Articles’ search

Articles’ selectionArticles’ selection


Statistical analyze

Introduction Objectives AcknowledgmentsResultsMethodsMethods

Each selector gets a list of articles (of one of the periods)

Guided by a sequence of random numbers (ex: 36, 6, 235,…), he choose and open one article at a time

Random selection of articles



Reading and analyze of the abstract

The abstract is read and analyzed with the inclusion/exclusion criteria

Inclusion- report a study of diagnostic accuracy - the diagnostic test(s) is(are) used

in obstetrics and/or gynecology - abstract is available

Exclusion- is a clinical trial, a letter, an editorial, a

practice guideline, a meta-analyse or a review


Articles’ search



Statistical analyze


Decision

If excluded- the reason of exclusion is registered

If included or doubted- the abstract is reanalyzed by other

selector

When disagreement or doubt, the article was discussed by the two selectors. When no decision could be achieved, a third selector was consulted

PMID of each opened article is registered


Articles’ search



Statistical analyze


Final samples

20052005

20042004

2000-20032000-2003

1990-19991990-1999

1980-19891980-1989

10 10 articles articles samplesample



10 10 articlesarticles samplesample

10 10 articlesarticles samplesample

Global sample Global sample of 50 articlesof 50 articles

This is repeated until the getting of a sample of 10 articles for each period


Articles’ search



Statistical analyze

Articles’ search


Articles’ evaluationArticles’ evaluation

Statistical analyze


Section and topic Item Describe Value

TITLE/ABSTRACT/KEYWORDS

1 The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity') 1

INTRODUCTION 2The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups 1

METHODS 1

Participants 3The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected 1

4Participant recruitment: was this based on presenting symptoms, results from previous tests, or the fact that the participants had received the index test(s) or the reference standard? 1

5Participant sampling: was this a consecutive series of patients defined by selection criteria in (3) and (4)? If not specify how patients were further selected. 1

6Data collection: were the participants identified and data collected before the index test(s) and reference standards were performed (prospective study) or after (retrospective study)? 1

Reference standard 7 The reference standard and its rationale 1

Test methods8

Technical specification of material and methods involved including how and when measurements were taken, and/or cite references for index test(s) and reference standard

0,50,5

9

Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference standard

0,50,5

10

The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference standard

0,50,5

11Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other test(s) and describe any information available to them

0,50,5

Statistical methods 12Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals) 1

13 Methods for calculating test reproducibility, if done

0,50,5

STARD checklist


Participants 14 When study was done, including beginning and ending dates of recruitment 1

15Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s), comorbidity, current treatment(s), recruitment center)

1

16

How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the reference standard; describe why participants failed to receive either test (a flow diagram is strongly recommended)

1

Reference standard

17 Time interval and any treatment administered between index and reference standard 1

18Distribution of severity of disease (define criteria) in those with the target condition; describe other diagnoses in participants without the target condition

1

Test results 19A cross tabulation of the results of the index test(s) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard

1

20Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result and how they were handled

1

21 Adverse events of index test(s) and reference standard 1

Estimation 22Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals)

1

23Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done

1

24 Measures of test reproducibility, if done0,50,5

DISCUSSION 25 The clinical applicability of the study findings 1


Section and topic Item Describe Value

RESULTS

Participants 14 When study was done, including beginning and ending dates of recruitment 1

15Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s), comorbidity, current treatment(s), recruitment center)

1

16

How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the reference standard; describe why participants failed to receive either test (a flow diagram is strongly recommended)

1

Reference standard

17 Time interval and any treatment administered between index and reference standard 1

18Distribution of severity of disease (define criteria) in those with the target condition; describe other diagnoses in participants without the target condition

1

Test results 19A cross tabulation of the results of the index test(s) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard

1

20Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result and how they were handled

1

21 Adverse events of index test(s) and reference standard 1

Estimation 22Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals)

1

23Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done

1

24 Measures of test reproducibility, if done0,50,5

DISCUSSION 25 The clinical applicability of the study findings 1

STARD checklist (2nd part)


Articles’ search



Statistical analyze

Each item value 1 point

Items 8, 9, 10, 11, 13 and 24 concern index tests as well as the reference standard, so each one of these components was counted with 0.5 points


Groups

Article’s components

- Title,

- Abstract,

- Keywords

- Introduction

- Methods

- Results

- Discussion

3 Groups

2 evaluators (each analyze 50 articles)

4 evaluators

(each analyze 25 articles)

4 evaluators (each analyze 25 articles)

Each evaluator works independently

Mismatch results are analysed by a fourth group


Articles’ search



Statistical analyze


Databases

Articles’ search



Statistical analyzeStatistical analyze

Two databases were created

Database of selection

Contain

- PMID of the article

- Period to which the article belong

- Decision of the 1st selector

- Cause of exclusion presented by 1st selector (for the excluded articles)

- Decision of the 2nd selector about the articles included by the 1st selector

- Cause of exclusion presented by 2nd selector (for the excluded articles)



Databases

Articles’ search




Two databases were created

Database of evaluation

Contain

- PMID of the article

- Name of the article

- Name of the journal where the articles was published

- Publishing year of the article

- Impact Factor of the journal when the article was published

- Evaluation points given by one evaluator for each item

- Evaluation points given by the other evaluator for each item

- Final score for each item



Analyses

Articles’ search




- The frequency of each reason that caused exclusion of articles

- The probability of the article not be excluded (included or doubted) by the 2nd selector if it had being also not excluded by the 1st one

- Total STARD score for each article

- For each item in the STARD-statement, the total number of articles reporting that item

- Mean and standard deviation of the total STARD scores for each period

- Compare means (One-Way ANOVA method)

- Agreement between the evaluators – the reproducibility (Bland and Altman method & Intraclass Correlation Coefficient)

- Relation between articles’ STARD scores and impact factor of journals


Introduction Objectives AcknowledgmentsResultsResultsMethods

From articles’ search

From articles’ selection




From articles’ From articles’ searchsearch

From articles’ selection

Period Number of articles founded

2005 1232

2004 1169

2000-2003 4057

1990-1999 6368

1980-1989 2221

Total – 15047



Decisions of selectors


From articles’ From articles’ selectionselection



Causes of exclusion



From the 1st selector

From the 2nd selector

93

(74,7%)

1

(0,8%)

3

(2,4%)

1

(0,8%)

19

(15,2%)

8

(6,4%)

7

(77,8%)

1

(11,1%)

1

(11,1%)



Agreement between selectors



probability of the article not be excluded by the 2nd selector if it had being also not excluded by the 1st one Number of articles not excluded by the 1st selector

Number of articles not excluded by both selectors

108

990.92

92 %92 %



Final selection states




Introduction Objectives AcknowledgmentsAcknowledgmentsResultsMethods

• Altamiro da Costa Pereira, MD,PhD

• Cristina Santos, MD, PhD


Acknowledgments

Documents

The reporting of diagnostic accuracy in obstetrics and gynecology literature an overview on the report quality between 1980 and 2005 : Class1 :Ana Costa,