COMPUTER SCIENCE DEPARTMENTV

V

Faculty Meeting - December 14, 1978V

Present: Feigenbaum (Chairman), Brown (Associate Chairman),Binford, Buchanan, Earnest, Herriot, Knuth, Lenat,McCarthy, Owicki, Wiederhold, and Student RepresentativesJim Davidson and David Wall. Absent: Baskett, Dantzig,Flynn, Golub, Green, McCluskey, Tarjan, vanCleemput, West ,Winograd, and Yao, F. On leave: Floyd, Oliger,Wilkinson, Yao, A.

Degree Candidates

The degree candidates for winter quarter were:

Armando Rodriguez, Master of ScienceRichard Giusti, Master of Science, Computer EngineeringWendy Peikes, Master of Science, Computer Engineering

They were voted on, contingent upon approval of the ComputerEngineering Committee for Richard Giusti and Wendy Peikes and uponcompletion of degree requirements for all three. Approval wasunanimous.

Conway Report

The Conway report of 72 computer science departments in NorthAmerica was published by Cornell. Stanford is still #1. BruceBuchanan suggested printing this news in the Campus Report.

National Academy of Sciences

Ed announced two requests from the National Academy ofSciences for Eastern European scientists who wanted to visitStanford. Although visiting scholars can be handled, on theunderstanding that they cannot have office space or computer time,the hosting responsibilities for a foreign visitor are cumbersome.Gio said he was interested in Dr. Palac Jozo. John McCarthysuggested keeping some of the CSD space for visitors when we moveto Margaret Jacks Hall. Denny said that almost all the CSD spacehas been earmarked for other uses.

The meeting came to order in Polya 204 at 12: 15

Action: Ed will call the NA.S and refer them to Gio for Dr.Jozo.

NSF Brochure on Submitting Proposals

NSF has sent a brochure inviting researchers to submitproposals for support. This will be xeroxed and circulated amongthe faculty.

Subcommittee Meeting in Washington

Ed gave a report of the meeting in Washington of the NSFAdvisory Committee on Computer Science. They are preparing atestimony for Congress to increase support of research in computerscience"

DEC 2060

In a discussion of the DEC 2060, the new department computer,Ed asked that the various projects set aside some money in theirbudget to buy a share of the computer, on an annual basis, to helppay for it. The computer will be delivered on January 8, withinstallation to be completed on January 15. Tnree fourths of thememory will not be delivered until March or so.

Move to New Building

The faculty agreed that they would rather make the move toJacks Hall as soon as possible, and get it over with, than waituntil summer.

Les Earnest brought up the matter of the telephone system inthe new building. Suggestions on how it should work ranged fromissuing beepers to everyone to equipping every office with fivetelephone lines.

The student bureaucrats favored paging in the lounge. Onlyone faculty member did not want paging in the lounge. Otherfaculty members were neutral.

For current details, see Les Earnest's draft telephone plan.

The meeting adjourned at 1:35

Respectfully submitted,

IV Pwv^ I Wx? oV^-o-*—

Mary McGoveran

completionCommittee

requirement

DECREE CANDIDATES

AUTUMN1978/79

DEPARTMENT OF COMPUTER SCIENCE

MASTER OF SCIENCE

Rodriguez, Armando

MASTER OF SCIENCE - COMPUTER ENGINEERING

The Computer Engineering students are presented contingent uponof degree requirements and approval of the Computer Engineering

Mr. Pvodriguez is presented contingent upon completion of degree

Giusti, Richard W.Peikes, Wendy

December 14, 1978

<

A SURVEY OF GRADUATE PROGRAMS

IN COMPUTER SCIENCE

by

Richard Conway

TR7B-357

Department of Computer ScienceCornell UniversityIthaca, New York 14 85 3

V

!l

This report presents the results of a survey of computerscience graduate programs conducted in September and October of1978. A questionnaire was sent to the department chairman ateach of the seventy-one "Universities Offering Ph.D. programs inComputer Science", based on the 7-13-78 list from the SnowbirdConference. This list turned out to include several universitiesthat do not, in fact, offer a Ph.D. in computer science and mayhave omitted some that do, but itthe study.

nevertheless was the basis of

Each chairman was invited toseventy-one programs with respectusing the following scale:

rate (not rank) each of theto "quality of graduate faculty" ,

5 for "distinguished"4 for "strong"3 for "good"2 for "adequate"1 for "marginal"0 for "not sufficientblank for "inadequate information to rate"

They were also asked to rate each institution with respect to"effectiveness of doctoral program", using the following scale:

3 for "extremely attractive"2 for "attractive"1 for "acceptable"0 for "not attractiveblank for "inadequate information to rate

They were also asked to estimate the first derivative of each pro-gram, using the following symbols:

+ for "gaining in quality"0 for "not changing"

for "declining in quality"blank for "inadequate information to rate

This terminology was borrowed from the Roose-Andersen "Ratingof Graduate Programs" for the American Council on Education (1970) ,but there the similarity ends. That study was broadly based andcarefully controlled—this one just went to chairmen of the programsbeing rated. The Roose-Andersen study can be criticized as favoringprograms with many graduates; this study perhaps reduced that risk,

"

but surely introduced many others.

In addition to ACE-like ratings, each chairman was asked to"list institutions that you would recommend as having especiallystrong programs in" seven different subject areas within computerscience:

1. Theory of computation, complexity, algorithms

2. Programming systems, languages and methodology

3. Operating systems

4. Machine organization and hardware systems

5. Numerical analysis

6. Database systems

7. Artificial intelligence.

There are many reasons why these results ought not to be takentoo seriously. To begin with, I made no attempt to achieve con-sistency of interpretation by trying to define "quality" or "effect-iveness", or the descriptors "distinguished", "strong", "attractive",etc. I did not suggest how the ratings should be produced. Itwas^apparent from the letters that some responses were the chair-man's personal opinion, some represented a systematic departmentalconsensus, and in at least one case the chore of replying was dele-gated to an unfortunate assistant professor. I did not attempt todefine what should be included in a "program" to be rated. Atmany institutions some relevant faculty may be in mathematics orelectrical engineering departments, as veil as in the nominal com-puter science program. Many schools also have research instituteswhich contribute strength to a graduate program. But I left allthese uncertainties to the respondent's discretion.

The results should also' be suspect just because the study wasconducted by an individual at one of the institutions being rated.

Such

studies, if they ought to be done at all, should be carriedout by some impersonal, objective organization. But such organiza-tions do not respond quickly, and I felt that the virtue of imme-diate action at least partially offset the effrontery of individualeffort. Cornell's rating may have suffered from the respondents'resentment of my initiative, or it may have benefited from embarass-ment at letting me see how little they really thought of our program.

For all these reservations, the results are undeniably inter-esting, and even schools that declined to participate may not beable to resist looking at a summary of their peers' opinions.

Thirty-four responses were received. Surprisingly, most re-sponses rated most of the programs, and more than half of the re-sponses rated all the programs.

Thirty of the responses included a rating of their own program—which provides material for an interesting substudy in itself. Noschool underestimated the esteem with which it is regarded by peerinstitutions. The extent of overestimate, in comparison to the

average of peer institution ratings, varied from 0.03 to 2.90(on the 5-point "quality" scale) . The largest difference was bythe school which was ranked 70th by average quality rating, andwhose self-rating would have put it in 7th place. Curiously,highly rated schools seemed almost as optimistic in their self-evaluation as lower rated schools. All self-ratings arc omittedfrom the data given below, '

Apart from self-ratings, the responses were remarkably con-sistent. There was rarely more than a one point range in thequality rating of a given program in different responses.

I have attempted to present the data so that it is not possibleto identify which schools participated in the survey, much lessdetermine what individual responses were.

However,

I should notethat the responses generally came from the schools that turned outto be relatively highly-rated by the study. This suggests a cer-tain circularity in the process, and you should be aware of thispossibility as you examine the results.

Quality of Graduate Faculty

The primary result of the survey concerns the quality of grad-uate faculty as perceived by other institutions. The average ofthese ratings is given in Table 1. Fifty-three schools had'an aver-age rating of 2.00 or more, corresponding to "adequate" quality.These fifty-three schools arc ranked by average perceived quality,and divided into four groups with breakpoints of 3.50, 3.00, and2.50. The detailed ranking is shown only for the first group; theother groups are listed alphabetically.

Table 1 also shows the "net derivative" for each of theseschools. If the number of responses indicating a school is "gainingin quality" exceeds the number indicating "declinina in quality",the net derivative is shown as a "+"; if the opposite is 'true thenet derivative is "-". If the number is equal, a "0" is shown.

For twenty-six schools the net number of raters (differencebetween "gaining" and "declining") exceeded 15% of the number ofresponses that gave a derivative for that school. These schoolsare listed in four groups in Table 2. Note that these listings in-clude several schools that do not appear in Table

1,

Recall that the questionnaire elicited the sign, but not themagnitude of the derivative. Consequently, for a school to aooearin the +30% group does not necessarily imply that the raters thoughtit was gaining more rapidly than a school in the +15% group—onlythat the progress is more widely recognized. The same is true forthe "declining" groups.

j

■f

Wash (St. I.oula)

Table 1 Average Perceived Quality

of Graduate

Faculty.

Average Subject AreaQuality cot Recoraiendations (Table 3)Rating Derivative Max Croup Nbr Subjects

Group Ql: 10 schools 5-3.5(ranked by avg)

1.

Stanford

4.97 0

SI

7»" MIT 4.76 si 73. Carnegie-Mellon 4.70 -

S2

6

4.

Cornell

4.52 +

SI

5 (J)5. Cal-Berkeley 4.47 +

£2

66. Illinois 4.05 -

SJ

5

7. Toronto

3.91;

+ " s4 6 (5)8.

UCLA

3.69 + E5 39. Purdue 3.65 -

£4

3

10. Waterloo 3.50 +

S4

4 (3)

Group 02: 9 schools 3.5-3.0(alphabetical)

Maryland_

s5 . ,-.Michigan_

s5 .KYU

_E5 I

Princeton -

S4

3 (2)Texas . s4 6Utah

_E5 2

Washington +

S5

3Wisconsin

_s4 „ (J)Yale

S5

5 (4)

Group 03: 16 schools 3.0-2.5(alphabetical)

Brovm +

S6

2 (0)Cal-Irvine „ lU '

Cal-Santa Crui

0

S6

1 (o)

Cal

Tech + ~ 3J~7^Harvard _, * "'Massachusetts_

« , i '.

£6

1 (0)

Minnesota 0 s6 . , . .North

Carolina

0 S6 l ,*(northwestern 0 \\ \ |«

Ohio

state -

'

£5 1Perm

State

„Pennsylvania . %l % jj

!,:>

2 (1)

Rochester 4

.m

SUNY-Buffalo f? * ,„ .SUNY-Stony Brook + . x l0)

VSC

*

£5

3 (2,

Group Q4: 18 schools 2.5-2.0(alphabetical)Arizona +British Columbia + E5 ,

Cal-Santa

Barbara + -, f , ,£6 1 (0)

Case-Western

Colorado

.

_Columbia

I

S6

1 W

Duke 0Georgia Tech +Indiana 4. _, . ,+

S6

1 (o)

lowalowa

State

qKansas +

Michigan Etato 0Pittsburgh 0Rutgers *

,

. +

£6

I (0)

SyracuseVirginiaSyracuse rrVirginia £63 {0)

65

The subject area recommendations provide an interesting checkon the rationality of the general quality ratings. Both the maxi-mum group and number of subjects rather strongly support the rank-ing by average quality rating.

Table 2 Perceived Direction of Change

Net Percentage of Responses indicating:Table 3 is also instructive with regard to the state of thevarious subject areas. For example, the "systems" side of computerscience does not enjoy nearly as clear a consensus as to which' arethe strongest programs as does the "theory" side. (It was also

remarkable how many schools recommended themselves in systems butdid not have this recommendation seconded by any other institution.)The scattered recommendations in database systems are not surprising,considering how recently this subject has claimed the attention ofcomputer science departments, but the relatively scattered recommend-ations in numerical analysis were unexpected.

"Gaining in Quality">

30%

> 1"Declining in Quality"

> 15% > 3> 15% > 30%

Arizona Brown Harvard Case-WesternOhio

State

Cal-S Barbara Colorado lITCal Tech Columbia Massachusetts PermRochester Delaware Michigan PermUSC Georgia Teoh WisconsinWashington Indiana Yale

Perm

State

Kansas

ÜBC

VPI Effectiveness of Doctoral ProgramWaterloo

(schools are listed alphabetically) Following the pattern of the ACE study, the questionnaireasked for ratings of the "effectiveness of the doctoral program"on a three-point scale. Forty-eight schools had an avcrage'ratingof^l.OO or greater, corresponding to "acceptable" on the Question-naire. These schools were ranked by their average effectivenessrating, and divided into four groups, as shown in Table 4.

The remaining columns in Table 1 are derived from the subjectarea recommendations and are explained in the next section.

In general, the effectiveness ranking roughly coincides with thequality ranking. However, five schools —Colorado, Minnesota,North Carolina, Syracuse and Wisconsin —ranked significantly higherin perceived effectiveness than in perceived quality of faculty.

Subject Area Recommendations

The recommendations of "especially strong programs" in sevensubject areas of computer science are summarized in Table 3.Schools are grouped according to the percentage of responses inwhich they were recommended. All self-recommendations were ex-cluded from the data.

Responses from Ql, Q2 and Q3 Schools

Groups SI to S5 are in equal increments of 20%. Group S6was added to segregate schools that were mentioned on only one re-sponse. Recall that the questionnaire did not suggest how manyschools might be recommended in a given area, and there was sig-nificant variation between responses in this regard. There wasalso variation between different subject areas. Note that the re-sponses were not asked to rank the schools in a given subject—only to recommend or not. Consequently, the percentages in Table3 indicate only how widely a school is well-regarded, and not howhighly it is regarded in comparison to other schools.

Many people suggested that the schools strongly committed toresearch might have a different perception of quality than thecommunity in general. To test this supposition, I partitioned theschools based on the average quality ranking given in Table 1.Groups Ql, Q2 and

Q3

totaled thirty-five schools— just half thetotal list. These schools provided twenty-one of the thirty-fourreponses.

Table 5 gives the results with respect to average cuality,if only the responses from the Ql,

Q2

and

Q3

schools (Table 1) areconsidered. The "net derivative" in Table 5 has the same interpre-tation as in Table 1. The rightmost column indicates the school'sranking in Table 5 relative to Table 1.

The results of Table 3 are summarized in Table 1 in two dif-ferent ways. The "Max Group" column indicates the highest per-centage group for any subject area. This might suggest how widelyrecognized the school's best subject is at other schools. The "NbrSubjects" column indicates in how many of the seven subject areasthe school was recommended. (A second parenthesized figure isgiven for the number of subjects if Group S6 is excluded.) Thisprovides some indication of the perceived breadth of quality.

The actual values of the average quality rankings in Table 5are interesting.

Overall,

the ratings are slightly harsher—onlyforty-four schools have an average rating of 2.00 or higher, ascompared to fifty-three in Table 1. Although there were interesting

4 V

4

"

r

■7'

"am

5-1.3

3Table 3

Pecorrffcndatlons of

Strong Programs by Subject

Schools

grouped by percentage

of

responseswere rccoT\-iended

in vhich they

{listed alphabetically vithin groups)

I Croup

SI

f Group

S2

f Group

S3

| CroupTable 4 Average Perceived Effectiveness of Doctoral Pro'

Group

S2

f Group

S3

( Croup

S4

Group

S5

Croup

S6

singloSubject (Schools marked * are ranked significantly higher thanfor Quality of -Graduate Faculty.)

mention

Theory,complexityalgorithms

Cornell

StanfordIllinoisMichigan

HarvardMinnesotaSyracuseUSC

Perm StateWashingtonWaterlooYale

Group El: 9 schools 3-2.0(ranked by avg)

1. Stanford 2.76

Wisconsin

Programing,languages,ccthodology

CarnegieMIT

CornellStanford

Waterloo

EerJceleyHarvard

Brown 2. MIT 2.72IndianaMarylandPerm

3. Carnegie-Mellon 2.69IllinoisMichigan 4. Cornell 2.53

5. Cal-3erkeley 2.45PrincetonSanta CruzSyracuse

6. Toronto 2.317. Illinois 2.228. Purdue 2.079. *Wisconsin 2.00

Yale

Operating Group E2: 9 schools 2.0-1.5BerkeleyMichigan

Stanford

Texas

Cal

TechCornellUNC

Group E3: 12 schools 1■ystens (alphabetical) (alphabetical)MarylandMichigan

*MinnesotaPrinceton

Cal TechColoradoHarvard

Perm StateS BarbaraWaterloo

TorontoUCLA

Yale Massachusetts♦North Carolina

WashingtonWisconsin Texas

UCLA

NYUWashington Ohio State

RochesterKachine Berkeley

Cal

Tech

Stanford

UCLA

Mass Waterloo

org.

,hardware

(^WesternTex a 3 Yale SUiN'Z-Stony BrookToronto *SyracuseWashington

USC

UtahNumericalanalysis DrowT.

SUNY-BufWisconsin Cornell

Group E4: 18 schools 1.3-1.0Maryland(alphabetical)Arizona

MIT

lowaBritish Columbia lowa StateBrown Michigan State

NorthwesternPerm State

Cal-IrvincCal-Santa Barbara

Database

Cornell

Maryland

Colorado

HarvardSyracuse

Cal-Santa

Cruz

PennsylvaniaRutger3

■ystensDukeGeorgia TechIndiana

SUNY-BuffaloWashington (St. Louis)State

PermPrinceton

Stanford

TexasUtah

Artificial MIT

Cal

TechPermintolll- Stanford

gence 'Chontr-r tutgerrSanta Cru:

j Wl urrnnstn

> 80» > 60» > 40t > 20» i 20%

Berkeley MIT CarnegiePrincetonToronto

TurdueTexasToronto

UCLA

USC

USC

Utahl

Illinois I BerkeleyTexas Cornegio

NYUPurdueTorontoWaterlooYalg

:hlgan

linoisMaryland

■xaa

9

*

i.

Table 5 Ajj^gj^e^erceiyed^uality of Graduate Faculty(based on Table 1 Ql 02 and Q3 reponses)

Net Change relativeto Table 1Deriva tive

0 samesamesame

+++

samesamehigher

lowerhighersame+

higherlowerlower

lowerlowerhigher+

+ samehigher

Group R3: 11 schools 3.0-2.5(alphabetical)Brown

higherlowerlower

HarvardMassachusetts

Minnesotahigher-higherlower

North CarolinaNYU

PennsylvaniaRochester lower

highersameSUNY-Stony Brook

USC higherlower-Utah

Group R4: 16 schools 2.5-2.0(alphabetical)

Arizona -It

loverhigherhigher

British ColumbiaCal-Irvine

Cal-Santa CruzCal TechColorado

Dukelowa StateNorthwestern

Ohio StatePerm StatePittsburgh

lowerhigherlower

higherlowerlower

lowerlowerhigher

Rutgers higherhigherlower

SUNY-BuffaloSyracuse

Washington (St. Louis) higher

AverageQualityRating

Group Rl: 9 schools(ranked by avg)

1. Stanford2. MIT3. Carnegie-Mellon

5.0-3.5

5.004.864.85

4.5.6.

CornellCal-BerkeleyToron to

4.604.504.00

7.8.8.

IllinoisYaleUCLA

3.853.553.55

iroup R2: 8 schools(alphabetical)

MarylandMichiganPrinceton

3.5-3.0

PurdueTexasWashington

WaterlooWisconsin

*

f

11'1?

local rearrangements in the list, the general pattern in Table 5is not much different from Table 1.

However,

eight schools —

Alberta,

British Columbia, Delaware,

Duke,

Hawaii, North Carolina,Washington (St. Louis) ar.d Yale are significantly higher-rankedin the Table 5 data.

SUNY-3uffalo and SUNY-Stony Brook from Q3, and similarly, manyschools from

Q4

. In fact, more than twenty-one additional schoolsmust be interleaved into Bitner's list to match Table

1,

so hesomehow selected fewer than half of the stronger schools for hisranking.

Table 6 summarizes the perceived changes in quality in amanner analogous to Table 2. It is interesting that Table 6 showsmore highly-ranked schools perceived as "declining in quality"than does Table 2.

Ranking from:Table 1 Table 5

1. Stanford2. Carnegie3 . MIT

01 1 P.l 1Ql 3 Rl 3A year ago James Bitner of the University of Texas asked the

chairmen of twenty-one computer science departments to rank thosetwenty-one departments. He distributed the responses in August,1973. His ranking is shown in Table 7, along with the correspondingrankings from Table 1 and Table 5.

Ql 2 Rl 24. Cornell Ql 4 Rl 45. Cal-3erkeley6. Illinois

Ql

5 Rl 5

Ql

6 Rl 7Q2 H27. Texas

8. Purdue9.

UCLA

p.:Ql 9 R2For the particular schools Bitner selected, there is general

agreement even though the rankings were produced by quite differ-ent processes. An exception is the University of Washington, whichfares much better in current study.

Ql 8 Rl 810. Yale Q2 Rl 8

11. Michigan12. Wisconsin13. Maryland14. Ohio State15. Harvard

02 R2R2R2R4P. 3P. 4P. 3P2R3R4

02The origin of Bitner's list of schools is not obvious. Incomparison with Table

1,

he omitted Toronto and Waterloo from

Ql,NYU,

Princeton and Utah from Q2,

Brown,

Cal-Irvine, Cal-Santa Cruz,Cal Tech, Massachusetts, Minnesota, North Carolina, Rochester,

02Q303

16. Perm State17. USC

0303

18. Washington

19,

Pennsylvania20. Northwestern

21,

Michigan State

02Q3Table 6 Perceived Direction of Chance 0304(based on Ql,

Q2

and

Q3

responses)

Net Percentage of Responses indicating: Conclusions

"Gaining in Quality"> 30% >

"Declining in Quality"> 15% > 30%> 15%

Arizona Alberta Cal-Irvine Case-WesternBrown Cal-Eerkeley Carnegie Ohio StateCal-S Barbara Columbia Harvard PermCal Tech Delaware Illinois Perm StateColorado Georgia Tech lIT TexasRochester UEC Michigan YaleUSC UCLA MITWashington Waterloo Purdue

UUhWisconsin

(schools aro liatod nlphoboticnlly)

Some people apparently feel that this type of information isnot useful, but at least it provides prospective graduate studentsand prospective faculty members with an interesting check on theprejudices of their local advisors. Others may feel that howeveruseful, such a study should be conducted only under the aecis ofsome objective professional society. I agree, and hereby retirefrom the survey field having, I hope, achieved the objective ofreplacing the Bitner ranking with a broader and fairer comparison.For the benefit of my successors, if any, I suspect it is possibleto invest a great deal more effort in such a study with only mar-ginal improvement in the utility or credibility of the results.

Table 7 "Bitner Ranking" of Selected Schools

Comparison to Previous Bitner Survey

-12-

Let me underscore once more that such a study. addresses onlyperceived quality. Eut the perceptions of other institutions arenot unimportant, I suspect many of us will discover from theseresults that wo have not done an adequate job of informing otherschools of our true distinction. In fact, several schools havealready embarked on an educational campaign. But note that thisseeks to chance the perception rather than the quality.

There is always the possibility that in a few isolated cases,the perception might be closer to the truth than self-evaluation.If just a few schools recognize that possibility and use this in-formation to justify strengthening their program, then this wholeexercise will have been worthwhile.

"i

i