TTN – WP3 TTN meeting, June 10-11, 2010

TTN meeting 10-11 June 2010 TTN /WP3 Marcel Soberman CNRS/IN2P3

Invention Discl/Year

0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network

TTN – WP3

TTN meeting,June 10-11, 2010



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network WP3 members

Name Institution Comment

Caccia Massimo INFN

Clerc Gabriel EPFL

Dargent Pascal IN2P3

Paolucci Lorenza INFN -> CERN

Parrinello Claudio CERN

Rüeck Dorothee GSI

Rudolph Robert PSI

Soberman Marcel IN2P3 convener

Stres Spela IJS

Surowiec Alicja GSI

Julie Bellingham SFTC

Wurr Karsten DESY to be replaced

Le Goff Jean-Marie CERN TTN mngr



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkWP3 - objectives and methodology

• To investigate, define and classify a set of criteria for TT activities measurement in PP

• How? To build a set of indicators and metrics– overview of the situation in our institutions– elements of comparison between us, and us vs overseas– guidance for newcomers– performance improvement measurement

How to select those indicators?– bibliography– adjustment to our research profile– testing using a questionnaire– define the final ones: the TT PKIs for HEP institutions

Presented last T

TN meetin

g

(December 2

009)



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkQuestionnaire – sent in April 2009

Presented last T

TN meetin

g

(December 2

009)



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkSchedule at end 2009

TTN meeting 10-11 December 2009

Integration of TTN meeting remarks January 2010

WP3 meeting - preparation of 2010 actions January 2010

Updated questionnaire sending was March 2010

Report preparation deadline for CERN council: July 2010

Booklet was July 2010

CERN council September 2010

Presented last T

TN meetin

g

(December 2

009)



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkQuestionnaire analysis: last TTN meeting issues

At the end of 2009, 19 ‘2008 questionnaires’ were received, 14 considered as valid (not too much empty fields), splitted in subsets: EG, EH (using FTE-HEP), EU, NG ALL vs ASTP, HEP vs a ENCHMARK=(BNL+EPFL)

• the questionnaire was designed to get answers on the HEP activity, but responses covered a mix between HEP and non HEP

• empty cells are significant – for ex. we can show that facilities agreements are indicated only in HEP institutions-, but calculations are disturbed

• possible misunderstandings in the answers• very large variance • results sensitive to the quality of data and the choice of selected

questionnaires in each set• not the same indicators than ASTP better identification of homogeneous subsets add other questionnaires other ways of calculations (mean on significant variables by reducing the

impact of empty cases…) consolidation of KPI choice



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkMajor evolutions since previous TTN meeting

• WP3 meeting (22/01/10 in Paris), with main decisions:– selection of KPIs for the analysis and for future questionnaires– distribution in two subsets: HEP institutions and ‘BENCHmark

institutions’ (having high performance in TT)– report and booklet structure preparation– new schedule with the objective to add more questionnaires to be more

confident on the results: that’s where the shoe pinches, because of delays in the receipt of new completed questionnaires!

• Only two new completed questionnaires were received (more were expected), classified as ‘Universities’ :– University College London (GB)– Politecnico Di Milano (It)

• Reorganisation in two groups:- HEP institutions (all facts considered only through HEP activities)- BENCHMARK: generic institutions having high performance in TT

[BNL (US), EPFL (S), UCL (UK)]



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network Work done in 2010

• 21 ‘2008 questionnaires’ were received, 16 considered as valid (not too much empty fields)

• Anonymisation of questionnaires, on the request of some institutions• Splitting in a first group of two subsets:

– ALL 2006 (16 Q) mixing HEP, multipurpose institutes and Universities– ASTP (Association of European Science and Technology Transfer Professionals)

survey added for comparison• Splitting in a second group of two subsets:

– 10 HEP institutions having a profile pure HEP (all facts considered only through HEP activities)

– BENCHMARK: 3 generic multipurpose institutions (existing HEP activity is not the measure) with high performance in TT [BNL (US), EPFL (S), UCL (UK)] – also active in TT but n

• Analysis:– Descriptive statistics– Selection of KPIs– Comparison on KPI means– Research of explaining factors in each subset– Comparison ALL vs ASTP– Comparison HEP vs BENCHMARK– Radar graphs



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkQuestionnaire analysis –methodology (1)

• 1 Input raw data from questionnaires to Excel

• 2 Preparation of the synthesis– Total # FTEs= FTE for general institutions and HEP FTEs for HEP

institutions– Quantification of qualitative data (particularly those relative to

maturity)– One worksheet splitted in two sets: ALL inputs (16 institutions) vs

ASTP 2006, to have a global vision– One worksheet splitted in two sets:

• 10 HEP European institutions having provided HEP specific data vs a BENCHMARK of 3 institutions



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network Selection of KPIs

• Two references (not KPIs):– #FTE and #TTO

• Eleven KPIs (* see comments next page)– 2.1.1 # invention disclosures/year– 2.1.2 # priority patent applications/year– 2.1.5 portfolio of patent families– 2.1.6 portfolio of commercially licensed patents– (missing in Q2008) total portfolio of licenses (including software and

know-how)– (missing in Q2008) license revenue/year– 2.2.1 # IP transfer or exploitation agreements/year– 3.1.1 # R&D cooperation agreements/year– 3.1.1.4 R&D cooperation agreements revenues/year– (incomplete in Q2008): licenses+services+facilities revenue/year *– 2.3.4 # startups still alive since 2000 (not really significant, but for

information).



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkComments on these KPIs

• Maturity of HEP institutions is an interesting KPI; it was evaluated through an aggregate built from various answers with more ore less weighting; unfortunately, as it is today this indicator only measures if written rules exist

• Revenues related to knowledge and technology transfer activities have two sources: – the commercialisation of IP comprising licensing, services, consultancy

and access to facilities;– and R&D cooperation comprising collaborative and contract research

Revenuesfrom KTT activities

IP commercialisation

Licensing Services,Consultancy,Access to facilities

R&D cooperation(collaborative and contract research)

New IPResearchdisciplines

Products& GDP



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkQuestionnaire analysis –methodology (2)

• 4 Multiple correlation on aggregates: Search for explanatory factors– NB: empty cells have been set to zero; indeed, the Excel tool

cannot work on non numeric values

• 5 Normalised aggregates: aggregates are normalised to 1000 FTE equivalent per site, then all values are normalised between 0-1 for radar graphics and histograms

• 6 Comparison of means between each set of selected institution (normalised to 1000 FTS): to see where are the main differences and if HEP institutions are specific

• 7 Graphs criteria: a radar graph comparing all institutions on a selected KPI

• 8 Graphs Institutes: a radar graph on strengths and weakness of each institute



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6


Results – June 2010



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkDescriptive statistics

Our 16 relevant institutions represent:

• 68530 FTE (73339 if we include all questionnaires), on which 8043 FTE are devoted to HEP

• 142 TT officers

In 2008, they have produced:

• 540 invention disclosures

• 30 new startups – with 125 still alive

• 88 IP agreements

• 720 R&D contracts

• 159 M€ revenues from R&D contracts



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkALL selected institutions vs ASTP per 1000 FTE

Ratio ALL/ASTP

0,00

0,50

1,00

1,50

2,00

2,50

3,00

The comparison of the figures resulting of the ALL questionnaires vs ASTP gives:-less TTO (75%)-quite the same invention disclosures per year-more licensed patents (maybe due the calculation on 1000 FTE and Top ten HEP institutes + 3 ‘BENCHMARK’ institutes)



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkHEP institutions vs BENCHMARK

Ratio HEP/BENCHMARK

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

1,60

1,80

2,00

In this comparison, we have HEP institutions compared a benchmark set (2 EU, 1 US), normalized to 1000 FTE in each institute:-less TT officers per 1000 FTE in the benchmark (prob. due to their large size)-a not too bad score in terms of licenses with only ¼ in terms of IP agreements-services & facilities are specific to some HEP institutions (vs no answer for the others)



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkKPI means analysis

• The objective is to compare KPI means of of institutions vs ASTP• In a first time, we have a look on the KPIs of each set vs one other to observe if there are

interesting variations, to put the focus on them• The means are listed below:

Comparison on means FTE #TTO InventionDiscl/Year

Priority patents app./Year

Portfolio patent families

Portfolio Commercially Licensed

#IP agreements/Year

#Research agreements/Year

R&D contract revenues(M€)/Year

Service & facilities Revenues (M€)/Year

Startups alive

ALL 3 738 8 34 16 121 61 6 45 10,536 0,491 8HEP 813 8 10 5 30 8 4 34 11,519 0,785 6BENCH 4 809 21 77 33 208 89 11 160 46,333ASTP 1 000 8 17 6 6 6 109

Comparison on means with normalisation on 1000 FTE

FTE #TTO InventionDiscl/Year




#IP agreements/Year




Startups alive

ALL 4 2 9 4 32 16 1 12 2,819 0,131 2HEP 1 10 12 6 37 10 5 41 14,161 0,965 7BENCH 5 4 16 7 43 18 2 33 9,635ASTP 1 8 17 6 6 6 109

Comparison on means with normalisation on 1000 FTE 0-1

FTE #TTO InventionDiscl/Year




#IP agreements/Year




Startups alive

ALL 0,78 0,22 0,54 0,62 0,75 0,89 0,23 0,11 0,20 0,14 0,30HEP 0,17 1,00 0,73 0,94 0,85 0,56 0,84 0,38 1,00 1,00 1,00BENCH 1,00 0,43 0,96 1,00 1,00 1,00 0,35 0,31 0,68ASTP 0,21 0,82 1,00 0,93 0,00 0,35 1,00 1,00



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6


KPI means analysis per 1000 FTE

Comparison on means

0,00

0,20

0,40

0,60

0,80

1,00

1,20

ALL

HEP

BENCH

ASTP



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network Comparison on means

• Preliminary remarks:

– the normalisation of each institute on 1000 FTE improves the results for ‘HEP institutes’, particularly for those <<1000

– the results are not for all HEP institutes but for the top ten in TT

• If the mean number of TT officers between subsets can be compared, it’s very variable from one institute to another

• HEP invention disclosures and priority patents are satisfactory, with a good result in patent portfolio and patent licensing…but for CERN, GSI and STFC

• Contracts: the number of R&D contracts is difficult to appreciate independently of their amount, but we have very good results in terms of revenue, thanks to GSI

• Service and facilities revenues of some HEP institutes represent an interesting result, and will be grouped with license revenues in next questionnaires



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network Explaining factors

• Multiple correlation analysis has been used to measure the impact of each KPI on the others

• The threshold to consider if a high correlation exists has been chosen to 0,707 (see next figures) considering freedom=6 and confidence p=5%



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network ALL QTTN

In term of explaining factors, we have to suppress trivial correlations (patents # vs invention disclosures is an example).

Possible links:• # invention disclosures and # off TTOs• # startups alive and # patents

All TTN QTTN (without ASTP) FTE #TTO Written rules InventionDiscl/Year




#IP agreements/Year


R&D contract revenues(K€)/Year

Service & facilities Revenues (K€)/Year

FTE 1,00#TTO 0,63 1,00Written rules 0,21 0,15 1,00InventionDiscl/Year 0,86 0,78 0,16 1,00Priority patents app./Year 0,81 0,69 0,01 0,97 1,00Portfolio patent families 0,94 0,54 0,26 0,88 0,83 1,00Portfolio Commercially Licensed 0,91 0,43 0,16 0,85 0,83 0,98 1,00#IP agreements/Year -0,20 -0,06 0,11 0,13 0,18 -0,04 -0,02 1,00#Research agreements/Year -0,11 -0,09 0,31 0,15 0,23 0,07 0,06 0,63 1,00R&D contract revenues(K€)/Year 0,04 0,25 0,37 0,35 0,36 0,15 0,10 0,56 0,86 1,00Service & facilities Revenues (K€)/Year -0,26 -0,33 -0,17 -0,26 -0,31 -0,18 -0,20 -0,02 -0,13 -0,18 1,00Startups alive 0,70 0,33 -0,27 0,59 0,61 0,74 0,78 -0,10 -0,24 -0,26 -0,24



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkExplaining factors HEP vs BENCHMARKCorrelation Factors in HEP FTE #TTO InventionDis

cl/YearPriority patents app./Year



#IP agreements/Year




Startups alive

FTE 1,00#TTO 0,51 1,00InventionDiscl/Year 0,26 0,42 1,00Priority patents app./Year 0,31 0,27 0,71 1,00Portfolio patent families 0,52 0,17 0,75 0,62 1,00Portfolio Commercially Licensed 0,37 0,10 0,71 0,70 0,93 1,00#IP agreements/Year 0,02 0,52 0,54 -0,10 0,22 0,02 1,00#Research agreements/Year 0,07 -0,12 0,39 0,84 0,25 0,36 -0,39 1,00R&D contract revenues(K€)/Year 0,42 -0,11 -0,35 -0,03 -0,23 -0,32 -0,48 0,21 1,00Service & facilities Revenues (K€)/Year -0,15 -0,38 -0,19 -0,43 -0,26 -0,36 0,09 -0,21 0,13 1,00Startups alive 0,46 0,64 0,00 -0,03 0,11 0,07 0,10 -0,35 0,08 -0,35 1,00

Correlation Factors in BENCH FTE #TTO InventionDiscl/Year




#IP agreements/Year




Startups alive

FTE 1,00#TTO 0,96 1,00InventionDiscl/Year 0,96 0,84 1,00Priority patents app./Year 0,79 0,59 0,94 1,00Portfolio patent families 0,68 0,45 0,87 0,99 1,00Portfolio Commercially Licensed -0,67 -0,85 -0,43 -0,08 0,08 1,00#IP agreements/Year -0,37 -0,61 -0,07 0,28 0,43 0,93 1,00#Research agreements/Year -0,48 -0,70 -0,20 0,16 0,32 0,97 0,99 1,00R&D contract revenues(K€)/Year -0,28 -0,53 0,02 0,37 0,52 0,90 1,00 0,98 1,00Service & facilities Revenues (K€)/Year#DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! 1,00Startups alive #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! 1,00

-In HEP, the relation between # research agreements and priority patents; could we say that patents are related toR&D agreements?-More interesting factors in BENCHMARK (high TT results), unless # of licenses vs # of TTOs with a high correlation between R&D contracts and patents ( objective for HEP institutions)

empty cells



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkRadar graphs

Radar graphs are used to give an easy way in comparing more than three axis of values at a first time, and to see the evolution of the results on each axis vs the others.

We have defined two categories of radar graphs:

• Graphs criteria: a radar graph comparing all institutions on a selected KPI; by this way, each institution may compare its results vs the other ones– NB: values are normalised on 1000 FTE per institution and values between 0 to 1

to facilitate comparisons

• 8 Graphs Institutes: a radar graph on strengths and weakness of each institute, to know where to put efforts– NB: values are normalised on 1000 FTE per institution and values between 0 to 1

Following figures are shown as examples. Each institution having answered the questionnaire will received their full set.



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkGraph ‘Criteria’ for HEP institutions

Example of Graphs ‘Criteria’ (performance of each institution per kpi)-Radar graphs show that each institution is specific and may have strengths &

weaknesses-The high performance obtained by some institutions must be regarded as an objective

by others, and improved each year

InventionDiscl/Year

0,000

0,167

0,026

0,5340,291

0,0370,112

0,000

0,259

0,000

0,100

0,200

0,300

0,400

0,500

0,600EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkGraph ‘Institute’ compared to another institutions

Example of Graphs ‘Institutes’ (Strengths & weakness per Institution) for two institutions:

These graphs show where are the weakness of your institution, and where you have to work with the institution management...for better results next year.

EH5

0,167 0,280

0,1120,084

0,2330,1680,558

0,0050,004

0 ,0 0 0

1 ,0 0 0

FTE

#TTO

InventionDiscl/Year


Portfolio patent familiesPortfolio Commercially

Licensed

#IP agreements/Year

#Researchagreements/Year

R&D contractrevenues(K€)/Year



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkReport and booklet structure

Booklet structure (In italic, chapters pasted from the report)• 1 Purpose• 2 Scope and methodology of this survey• 3 Indicators selected (and meaning)• 4 Analysis and results• 5 Recommendations for improvement• 6 Future plans• 7 Summary of conclusions

Distribution• CERN Council • PP institution Directors• Policy makers• TTN members• Other comparable networks• European Commission?• Specific distribution to questionnaire senders with added figures

Presented last T

TN meetin

g

(December 2

009)



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT NetworkWork still to do

• Report to the council

• Booklet

• New questionnaire



0,000 0,0160,049

1,0000,546

0,0700,210

0,000

0,485

0,000

0,200

0,400

0,600

0,800

1,000EG5

EG1

EH1

EH2

EH3EH4

EH5

EH6

EH7WP3CERN MS TT Network Next steps

TTN meeting 10-11 June 2009

Integration of this TTN meeting remarks

WP3 meeting - preparation of second half 2010

Updated questionnaire sending

Report preparation

Booklet

CERN council September 2010

Documents

TTN – WP3 TTN meeting, June 10-11, 2010