Eft/New frontiers in evaluation Conf. April 2006 1 fteval - New Frontiers in Evaluation Conference April 24th-25th 2006 Vienna - Austria Laurent Bach,

1Eft/New frontiers in evaluation Conf. April 2006

fteval - New Frontiers in Evaluation ConferenceApril 24th-25th 2006 Vienna - Austria

Laurent Bach, Nicolas Carayol*, Patrick LlerenaBETA L. Pasteur University of Strasbourg and CNRS (UMR 7522)*ADIS Université Paris [email protected]

Current reforms in the French evaluation system :

the growing role of indicators and measurement of PROs’ TT

activities


OUTLINES

Reforms of the French public research systemLaw on Innovation 1999, various gvtal measures 2000-2005, Law on research March 2006Focus on S-I relations / TT

2006 1st implementation of New Law on Budgeting Procedures based on target / performance evaluation

Indicators of TT activities of public

research actors

2005/06 : 1st comprehensive study on TT activities of French universities (BETA-

April 2006)


OUTLINES

• the TT activities of French university : first results of the Beta study• what can we learn from it about the performance indicators implemented by the New Law of Budgeting ?

1. The context : French public research system : main features,

recent trendsNew law on budgeting : performance indicators

2. Study on the TT activities of French universities

3. An assessment of the indicators system of the New Law on Budgeting


CONTEXT : French public research system

Main features of the French public research system•Extreme complexity

"double" duality (still prevalent) :Research : PROs (± 25) vs universities (81) … BUT "mix

labs"H.E. : universities vs Grandes Ecoles

"mixed" roles : PROs = funding agencies, programmes agencies, research operators

•Largely dominating the overall research systemStructure of fundings / of staff (civil servant status)

•Poor use of evaluation Lot of levels, lot of bodiesLack of coherence, of clarity, of impact on the system


CONTEXT : Recent trends

80s - Law on Innovation 1999 - governmental measures (innovation side) 2000-2005 - Law on research 2006

•Development of competition-based programmesPROs, Ministerial funds…National Research Agency (ANR)

•Fostering of Science-Industry relations and TT2006 law : approx. 2/3 of the "new" 2006-2010 funds + almost all measures

•Concentration of resources on a local/regional basis"top-down" clusters (Pôles de Compétitivité)H.E/Research grouping under "new" administrative structures (PRES, Réseau Thématique de Recherche Avancée...)

•Slow development of New Public Management approaches

Contractualisation, evaluation bodies…


CONTEXT : The LOLF

LOLF : Loi Organique sur la Loi de FinancePublic Budgeting procedures / Parliementary levelAdopted 2001 - first implemented 2005 for 2006 budgetObjectives : control of public money /actions

multi-year strategic planning more flexibility of funds allocation

•Gvtal action : very broad missions->programmes->actions•Performance targets and related evaluation system : objectives / performance indicators•Budget year N+1 voted by Parliament depending on evaluation of success in year N (+ 5 years horizon)


CONTEXT : The LOLF

University research and the LOLF :

mission "Research and Higher Education" (encompasses most of the public research activities, including the activities of PROs)

13 general objectives - 30 indicators (+ sub-indicators)research : 6 general objectives - 12 indicators

programme "Higher education programmes and university research"

various actions, including those related to research activities (broken down in 6 broad disciplinary fields + 1 interdisciplinary)


CONTEXT : The LOLF

Research-related objectives & corresponding indicators: 1

to produce highest international level scientific knowledge

scientific production of universities

share of international level publications by universities on the overall French scientific publications share of international level publications by universities on the overall EC scientific publicationsshare of international level publications by universities on the overall world scientific publications

scientific acknowledgment of universities

two-years lag citation index of universities publications

to develop dynamism and reactivity of university research

thematic reactivity of universities

share of international level publications by universities in the priority scientific fields (base : French scientific publications)share of international level publications by universities in the priority scientific fields (base : EC scientific publications)share of international level publications by universities in the priority scientific fields (base : world scientific publications)


CONTEXT : The LOLF


effectiveness of TT policy

share of universities in patent claims by French actors at INPIshare of universities in patent claims by French actors at OEB

efficiency of TT policy

share of revenues from IPR in the research-oriented resources of universities

intensity of relations with firms

share of industrial contracts (French and foreign firms, public and priavte firms) in the research-oriented resources of universities

to contribute to the development of attraction of French research

attraction of universities

share of foreigners in researchers, teacher-researchers, post-doc students and research support staff paid by the State or by universitiesshare of EC-origine foreigners in researchers, teacher-researchers, post-doc students and research support staff paid by the State or by universities

to contribute to the enhancement of national economy competitiveness and the TT of research results


CONTEXT : The LOLF


to contribute to the construction of ERA-European Research Area

rate of participation of universities in EC FP

rate of involvment as prime of universities in EC FP

to develop the provision of service by the university

amount of resources from provision of service

share of universities own resources coming from provision of service

share of articles co-published with EC members in the publications of universities

For PROs, most of these indicators are the same + some others are added depending on the field of research


STUDY ON UNIVERSITY TT : Presentation

•performed : 2005-2006 BETA Univ. L. Pasteur + French TTOs network CURIE

•for assembly of university presidents (CPU) and French Ministry of Education, Higher Education and Research (MENESR)

•survey by e-questionnaire filled out by universities' TTOs General organisation and functionning of TTOs

(status, expend. and resources, policy, forms of contracts, personal, etc)

Measurement of activities of TTOs

•situation as end 2004 + time-based variables 2000-2004first of a regular evaluation in the future

•sent to all universities (88, incl. INP) + 11 members of CURIE; base : 2003 Ministry data

The study on TT activities of universities


STUDY ON UNIVERSITY TT : Main results

Sample and representativity :74 answers (66 univ. + 3 INP + 5 others)Rate of return : 74.7 %

78% of the universities + INPbetween 71% and 100% according to the type of universitiesbetween 8 and 10 out of the 10 Top universities (18 out of Top 20) according to Ministry data on research-oriented resources, number of teacher-researchers, industrial contracts, IPR revenuessample mean ≈ poulation mean for those criteria

Not 100% answers on 100% questions => following results on sub-sample of respondants (not detailed here)

First results of the study



Creation of TT activities : more than 50% of univ. since 1999 (law on innovation), some very old ones

Size of TTOs : 263 Full-Time Equivalent; 3.9 FTE per university; 1.2 TTOs staff for 100 teacher-researchers (exl. PROS staff of university labs)Total 2004 expenditures 9.049 million €; 148 K€ per university; 2% of 2003 research-oriented resources

Organisation :in 86% of the case TTOs = department or internal service from the university (1/5 : the 1999 born SAIC)31 universities have at least two TTOs (2nd = very often external entity)54% of the TTOs employees are civil servantsHead : engineer (57% of univ), half have experience in private business.



Research contracts and services :2000-2004 : 766.8 M€ Average per university per year : 2.6 M€ (research contracts = 10 x service contracts) / 73 contracts

Results on "outputs"

0

50 000 000

100 000 000

150 000 000

200 000 000

250 000 000

2000 2001 2002 2003 2004



Patent (applying – French priority/EPO) :2000-2004 : 1104 patents = 3.3 patent per university per year(≈ 20% of univ.: no patent)

Patent (ownership – French + extension)End 2004 : 464 patents = 9.9 patents per university58% fully owned - 240 are extensions (European patent or national ones)

0

50

100

150

200

250

300

2000 2001 2002 2003 2004



Licensing 2000-2004 : 351 licenses awarded (1.1 license per year per university50% on patent and related know-how30% awarded to less than 3 years old university spin-off

Revenues from license 2000-2004 : 22.1 M€ ((90% on patent & related know-how) )average : 0.2 M€ per university per year

Other results on creation of firms, spin-off shareholding, "entrepreneurship" of civil servants, etc

End 2004 : 243 "active" licenses (5.2 per university)126 licenses are generating money (2.7 per univiversity)


LOLF vs TT study : Some elements on the relevance of indicators

per year on 2000-2004 ("flow" data) :•number of patent claims•number of copyright on software•number of licenses awarded per year•income from licenses per year (€)•amount of industrial contracts (€)2004 ("stock data")•number of patents owned•number of current licenses•number of licenses generating incomes

TT performance indicators from the study :


•patent indicators : what about patent claim property regime ? - 1


Performance as effectiveness : coverage of indicators

2000-2004 : on 1091 patent claims, 79% (859 patents) full or co-ownership

254

438

54

113

0

50

100

150

200

250

300

350

400

450

500

ownership co-ownershipwith PROs

co-ownershipwith otheruniversity

co-ownershipwith firms

70

22

161

0

20

40

60

80

100

120

140

160

180

PROs other university firms

ownership waive





0

50

100

150

200

250

2000 2001 2002 2003 2004

ownership co-ownership ownership waived Total





0

20

40

60

80

100

120

140

160

2000 2001 2002 2003 2004

co-ownership with PROs/university co-ownership with firmsownership waived to PROs/university ownership waived to firmsclaim related to firms





LOLF indicators : all ownership regime ? Patent invention vs ownership ?"share of patents" : net increase vs relative

importance ?increasing proportion of co-ownership between

university and PROs => may make the negociation of license with firms difficult


•Contracting activities indicators : what about distribution according to the different sources ? - 1



0%

5%

10%

15%

20%

25%

30%

35%

40%

2000 2001 2002 2003 2004

Ministry PROs Local authoritiesForeign (EC…) Associations Firms


•Contracting activities indicators : what about distribution according to the different sources ? - 2



LOLF indicators : industrial contracts compared to research-oriented

resources : decreasing share ? distribution of different resources ?

service contracts compared to the operating budget (with a certain % as objective), and not to any research-oriented resources.



performance as effectiveness : link between TT activities

• correlation bewteen TT activities ?

positive correlation : no displacement effects ?high : industrial contracts - patent claim

industrial contracts - licences awardedweak : industrial contracts - licences generating revenues

industrial contracts - revenues form license=> patent / licenses for money ? Signalling ? Secure collaboration ?

weak for software with all => separate activity ?



• Towards a first measurement of a "TT chain-link "PATENT CLAIM

Patent claim before 2000

patent claim 2000-04

1104 (220,8/year)

no decision

patents owned 2000-04

patents owned 31/12/04

464

OWNERSHIP

licenses awarded 2000-04

185 (37/year)

current licenses

31/12/04158

LICENSES AWARDED

licenses generating revenues 31/12/04

66

LICENSES GENERATING REVENUES

revenues from licenses 2000-04

REVENUES

22 055 634 €(4 411 426€/year)

5 131 224 €

2004

(a)

(b)

(A)

(B)

(B)

(B)



• Towards a first measurement of a "TT chain-link "

2000-2004, one license => 119 K€ as revenue (137 K€).

As for 2004one patent owned => 0.34 (0.35) license 42% (38%) of the licenses are generating

revenues, ie one patent => 0.14 (0.13) license generating revenues

revenue : 78 K€ per license (81 K€) = 11 K€ (11 K€) per patent owned

LOLF : no indicator on this



performance as efficiency

•Positive correlations between performance indicators and TT resources, but Inputs ? Outputs ?

•Low correlation between TT resources and IPR revenues

LOLF : no efficiencywhat is called efficiency = IPR revenues / research-

oriented resources



Universities differ : SizeScientific fieldsTT strategy, TT organisation, TT resources, etc=> TT performance : indicator one by one + "performance pattern"

aggregation issue : the heterogeneity of French universities

LOLF : not taken into account risk over-using benchmark analysisrisk to try to concentrate means on a few

universities

structural indicators ? reductions of gap …



0

10

20

30

40

50

60

70

80

90

100

min : 1

max : 95

3 own more than 80 patents

50% own less than 10 patentsmean : 18.6

Example of asymetric distribution of performance : patents owned



"average university" is far behind "frontier university"

Top universities perform differently on different activities

0%

10%

20%

patents owned95

current licenses43

licenses withrevenues

18

software /year6

patent claims/year19.4

licensesawarded/year

9.8

revenues fromlicenses/year

3.137 M€

industrialcontracts/year

4.246 M€


complementary remarks

•use of "ratio", "share of"…≠evaluation in valueratio "against" other components of the French

system (PROs, firms)•publications : social sciences, mix labs, 2 years lag•reactivity : priority thematics : too broad vs too narrow•contribution to the construction of ERA : number of EC projects => scattering of resources, transactions costs

•organisational performance ? networking ? learning ?…



CONCLUDING REMARKS

•understanding TT activities from universities :1st study => exploitation of data => future studies :

more data to fill some gaps"optimal" organisation, given : various contexts,

various activities (scope economies)" optimal" scale : critical mass ? return to scale ? cf same problem in production of knowledge

•LOLF : severe shortcomingskey question : connection with other components of

the evaluation system, notably at programme level (cf competition-based programmes) and at new structures level (cf new agencies)


THANK YOU FOR YOUR ATTENTION

Documents

Eft/New frontiers in evaluation Conf. April 2006 1 fteval - New Frontiers in Evaluation Conference April 24th-25th 2006 Vienna - Austria Laurent Bach,