Database Publication Practices

Surajit Chaudhuri

Microsoft Research

Coping with Growth

Issue: Higher submission rate We are not alone in facing this problem

– What are others thinking?– ACM SGB Task Force on the Impact of

Increasing Conference Submissions (Chair: Alexander L. Wolf) – next 4 slides

My analysis and some painful suggestions

Basis of SGB Task Force Data

ASE (1995-2004) DAC (1998-2004) DATE (2001-2005) EComm (1998-2005) FOCS (1990-2004) FSE (1993-2004) GLSVLSI (2002-2005) ICSE (1992-2005) ICSM (1999-2004) ISSTA (1989-2004) Middleware (2000-2004) MobiCom (1995-2004)

MobiHoc (2000-2004) MobiSys (2003-2004) PODC (1994-2004) PODS (1993-2003) POPL (1973-2005) SCG (1995-2004) SIGCOMM (1998-2004) SODA (1990-2005) SPAA (1995-2003) STOC (1991-2005) UML (1999-2004) UAI (1992-2004)

MobiCom (1995-2004)

MobiCom

0

50

100

150

200

250

300

350

400

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Year

# o

f p

aper

s

0%

5%

10%

15%

20%

25%

30%

Submitted

Accepted

Rate

Observations/Perceptions: Stature/Workload

Conferences with lowering acceptance rates receiving more polished papers on narrower topics

Pressure to publish in top venues– CS argued that conferences more important than

journals; now we are suffering for it– grad students expected to publish in top places

Between a rock and hard place– grow PC, but lose coordination– shrink PC, but lose quality of reviews

Vicious cycle– people agreeing to do more PCs (can’t say no)– people do less per PC

Most Radical Idea

Rethink the role of conferences– reduce importance w.r.t. journals– reduce number and increase acceptance rates– conference presentations derived from “best”

journal submissions (rather than vice versa)– make tenure evaluation based on quality of top

five papers, not number of papers Journals have better scale properties

– larger reviewer pool– less time pressure on authors and reviewers

My Analysis and Suggestions

(No implication for SIGMOD06)

Request to the Steering Committees

Experimenting with procedural changes– Observe– Make only one significant change for a while– Observe

Avoid second-order changes that do not address the pain point

Second Order Issues

Double-blind reviewing– Expected to impact on selection of “border-line”

papers – Many ACM SIGs follow

» SIGGRAPH, AAAI, SIGCOMM

Author Feedback– Good for “venting”– Tight timeline for reviewing makes it ineffective

Accommodating Growth

Acceptance rate– Acceptance rate is 15%, 20% or 39%?

» Do the math!

– Not lower than many other large conferences– Mandated changes seem unreasonable

Diversity and Narrowing of Topics– AAAI: “Big Ideas”, Tech papers, Abstracts– Independent conferences as tracks (like WWW)– Reuse journals as publication

Program Constraints– Presentation decoupled from acceptance– Posters and Plenary (old KDD style)

Impact on Reviewing Infrastructure

Choice of PC members crucial– Quality of Papers highly correlated with PC members

“Everyone is a PC member” – VLDB05: 610 reviewers

Large PC has likely to have high variability– VLDB05: 610 reviewers– But, otherwise 3-review load is too high

Suggestions (next 2 slides)– Short-cuts in reviewing process– Throttle the “flow” of papers

Reviewing Quality/Load

2-level PC (but reduce dependence on external reviewers) [like AAAI, SIGGRAPH]– Senior PC member nominates – Handles 4/5 papers– Good training for future PC experience

Early Rejection– If a paper gets 2 “weak” rejects, it is rejected without a

third review General Chairs should yield more time to reviewing

– Electronic PC requires it– Electronic Proceedings make it easy

We should learn/coordinate with broader CS community

Throttle the flow

No other ACM SIG engineers pipeline– SIGGRAPHICS or EuroGraphics– Mobicom or SIGCOMM – Takes motivation away to make serious changes

Selectively “break the pipeline”– VLDB-> ICDE– SIGMOD -> VLDB (except for roll-over papers)

Send the right (not polite) message– Weak Reject” vs. “Reject”– Very short reviews for bottom 10% - 20%

More Serious Issues beyond Publication Practices

Legacy of mid’80s – mid 90’s– Parallel DBMS, Query Optimization– Debate between deductive DB and OODB washed

away!

Jury is still out on the last decade? Today:

– Are there too many problem statements?– Everyone working on “personalized” problems– Many “fuzzy” problems with “fuzzy” yardstick for

solutions?– Worry: Weakening link to systems/hard engineering