Using Social Networks Methodology to Evaluate Research & Development Programs

Using Social Networks Methodology Using Social Networks Methodology to Evaluateto Evaluate

Research & Development ProgramsResearch & Development Programs

Franco MalerbaFranco MalerbaCESPRICESPRI

Luigi Bocconi UniversityLuigi Bocconi University&&

Nicholas S. VonortasNicholas S. VonortasCenter for International Science and Technology PolicyCenter for International Science and Technology Policy

and Department of Economicsand Department of EconomicsThe George Washington UniversityThe George Washington University

Joint AEA/CES MeetingsJoint AEA/CES MeetingsToronto, CanadaToronto, CanadaOctober 27, 2005October 27, 2005

PRIOR RESULTS ON IST-RTD PRIOR RESULTS ON IST-RTD NETWORKSNETWORKS

Prior Results Prior Results European IST RTD NetworksEuropean IST RTD Networks

The network of research collaborations has:The network of research collaborations has:

A self-organizing structure, dominated by A self-organizing structure, dominated by “hubs”, which are also key nodes in National “hubs”, which are also key nodes in National research networksresearch networks

A scale-free architecture at the thematic levelsA scale-free architecture at the thematic levels

Prior ResultsPrior ResultsEuropean IST RTD NetworksEuropean IST RTD Networks

European research is characterized by “small world” European research is characterized by “small world” connectivityconnectivity

Strong tendency of scientists to cluster around national Strong tendency of scientists to cluster around national communitiescommunities

Strong tendency to cluster with research disciplines and Strong tendency to cluster with research disciplines and within industrial sectorswithin industrial sectors

The funding structure has a strong influence on research The funding structure has a strong influence on research co-operationsco-operations


As a result of the new As a result of the new Integrated ProjectsIntegrated Projects and and Networks Networks of Excellenceof Excellence::

The density of links is higherThe density of links is higher The share of participants in the principal component The share of participants in the principal component

is higheris higher The average path length is lowerThe average path length is lower Large firms and research institutes are more dominant Large firms and research institutes are more dominant

as gate-keepers of collaborationas gate-keepers of collaboration Small companies are “crowded out” relative to FP5Small companies are “crowded out” relative to FP5


The IST RTD network as a whole has “small world” The IST RTD network as a whole has “small world” characteristics - but this is not true for each and every characteristics - but this is not true for each and every one of its programmesone of its programmes

FP6 is more likely than other research collaboration FP6 is more likely than other research collaboration frameworks to:frameworks to:

Connect universities and industryConnect universities and industry Connect different research themesConnect different research themes Include new Member StatesInclude new Member States Include key patent-holdersInclude key patent-holders Include SMEsInclude SMEs

CURRENT STUDY ON IST-RTD CURRENT STUDY ON IST-RTD NETWORKSNETWORKS

Towards an ERA for IST:Towards an ERA for IST:Overall ObjectivesOverall Objectives

Develop and apply a quantitative analytical framework Develop and apply a quantitative analytical framework for the assessment of the characteristics and for the assessment of the characteristics and performance of networks supported by IST RTD in performance of networks supported by IST RTD in FP5 and FP6.FP5 and FP6.

Analyze knowledge and partnership networks in Analyze knowledge and partnership networks in selected IST RTD domains, concentrating on network selected IST RTD domains, concentrating on network nature, topology, time evolution and effectiveness.nature, topology, time evolution and effectiveness.

Supplement quantitative information with some Supplement quantitative information with some qualitative information, and inter-organizational qualitative information, and inter-organizational networks with inter-personal networksnetworks with inter-personal networks

Towards an ERA for IST:Towards an ERA for IST:Evaluation QuestionsEvaluation Questions

How do the characteristics of the How do the characteristics of the IST-RTD IST-RTD partnership and knowledge networkspartnership and knowledge networks compare with compare with the characteristics of the the characteristics of the global partnership and global partnership and knowledge networksknowledge networks of IST-RTD companies and of IST-RTD companies and with the characteristics of the related global with the characteristics of the related global networks?networks?

How well are the companies participating in IST How well are the companies participating in IST RTD programs positioned in the global partnership RTD programs positioned in the global partnership and knowledge networks?and knowledge networks?


How effective are IST-RTD networks as How effective are IST-RTD networks as mechanisms for transmitting knowledge?mechanisms for transmitting knowledge?

Are the Integrated Projects (IPs) and the Are the Integrated Projects (IPs) and the Networks of Excellence (NoEs) creating Networks of Excellence (NoEs) creating leading “knowledge hubs”?leading “knowledge hubs”?

What makes these “knowledge hubs” What makes these “knowledge hubs” effective?effective?


To what extent does the prominent network status of To what extent does the prominent network status of certain IST RTD companies of clusters match the EU certain IST RTD companies of clusters match the EU technological leadership in certain areas?technological leadership in certain areas?

Are the global networks of selected “hub” companies Are the global networks of selected “hub” companies with extensive ICT supply chains represented in the with extensive ICT supply chains represented in the FP6 IST RTD?FP6 IST RTD?

Are the perceived national IST “knowledge hubs” Are the perceived national IST “knowledge hubs” well integrated into the FP6 network?well integrated into the FP6 network?

Selection of IST

technology domains

IST-RTDFramework Programme 6

Matching of IPC codes with technological domains

Patent examiners

Field experts

Matching of SIC codes with technological domains

INNET alliances

EP-CESPRI patents/citations

KNOWLEDGE NETWORKS (Ib, IIb, IIIb)

PARTNERSHIP NETWORKS

(IIa, IIIa)

PARTNERSHIP NETWORK Ia

Towards an ERA for IST:Towards an ERA for IST:Network TypesNetwork Types

IST-RTD partnership networkIST-RTD partnership network

IST-RTD knowledge networkIST-RTD knowledge network

Global partnership network of IST-RTD project participantsGlobal partnership network of IST-RTD project participants

Global knowledge network of IST-RTD project participantsGlobal knowledge network of IST-RTD project participants

Global partnership network akin to the E technology unitsGlobal partnership network akin to the E technology units

Global knowledge network akin to the E technology unitsGlobal knowledge network akin to the E technology units

Towards an ERA for IST:Towards an ERA for IST:Examined ProgramsExamined Programs

FP6 Thematic Areas Strategic objectives

eSafety of road and air transports

eHealth

Technology-enhanced learning and access to cultural heritage

Towards a global dependability and security framework

Networked business and governments

eInclusion Applications and Services for the Mobile User and worker

Cross-media content for leisure and entertainment

GRID-based Systems and solving complex problems

1. Applied IST research addressing major societal and economic challenges

Improving Risk management

Broadband for all

Mobile and wireless systems beyond 3G

Networked audiovisual systems and home platforms

Open development platforms for software and services

2. Communication, computing and software technologies

Embedded systems

Pushing the limits of CMOS and preparing for post-CMOS

Micro and nano-systems

Advanced displays

3. Components and micro-systems

Optical, opto-electronic, photonic functional components

Towards an ERA for IST:Towards an ERA for IST:Examined ProgramsExamined Programs

FP 5

FP6

Key Actions Thematic Areas

1. System and services for the citizen

2. New method of wok and electronic commerce

1. Applied IST research addressing major societal and economic challenges

2. Communication, computing

and software technologies

4. Essential technologies and infrastructures

3. Components and micro-systems

TA 1-2-3 FP6TA 1-2-3 FP6

Not SelectedNot Selected 115115 27,327,3

SelectedSelected 307307 72,772,7

Projects

Not selected27%

Selected73%

TA 1-2-3 FP6TA 1-2-3 FP6

Participants

Not selectedNot selected 13401340 21,821,8

SelectedSelected 48144814 78,278,2

Not selected22%

Selected78%

ParticipantsParticipants: counted once for every project they have participated in

By instrument(projects)

CA: Coordination ActionIP: Integrated ProjectNoE: Network of Excellence

SSA: Specific Support ProjectSTREP: Specific Targeted Research Project

TA 1-2-3 FP6TA 1-2-3 FP6

CA IP NoE SSA STREP

0

0,1

0,2

0,3

0,4

0,5

Not Selected

Selected

Organization Type

HE: Higher EducationIND: industry

REC: ResearchOTH: Other

TA 1-2-3 FP6TA 1-2-3 FP6

HE INDOTH

REC

0

0,05

0,1

0,15

0,2

0,25

0,3

0,35

Not Selected

Selected

Not SelectedNot Selected Large CompanyLarge Company 10321032 21.1521.15

SMESME 260260 21.1721.17

SelectedSelected Large CompanyLarge Company 38463846 78.8578.85

SMESME 968968 78.8378.83

SMEs and Large Enterprises

TA 1-2-3 FP6TA 1-2-3 FP6

SMELarge

Company

00,1

0,20,3

0,4

0,5

0,6

0,7

0,8

Not Selected

Selected

Correspondence between TAs, IPC Correspondence between TAs, IPC and SIC Codes…and SIC Codes…

Technology Classes: PatentsTechnology Classes: Patents

ClassClass descriptiondescription ClassClass descriptiondescription

11 Electrical engineeringElectrical engineering 1616 Chemical EngineeringChemical Engineering

22 Audiovisual technologyAudiovisual technology 1717 Surface TechnologySurface Technology

33 TelecommunicationsTelecommunications 1818 Materials ProcessingMaterials Processing

44 Information TechnologyInformation Technology 1919 Thermal ProcessesThermal Processes

55 SemiconductorsSemiconductors 2020 Environmental TechnologyEnvironmental Technology

66 OpticsOptics 2121 Machine ToolsMachine Tools

77 Control TechnologyControl Technology 2222 EnginesEngines

88 Medical TechnologyMedical Technology 2323 Mechanical ElementsMechanical Elements

99 Organic ChemistryOrganic Chemistry 2424 HandlingHandling

1010 PolymersPolymers 2525 Food ProcessingFood Processing

1111 PharmaceuticsPharmaceutics 2626 TransportTransport

1212 BiotechnologyBiotechnology 2727 Nuclear EngineeringNuclear Engineering

1313 MaterialsMaterials 2828 Space TechnologySpace Technology

1414 Food ChemistryFood Chemistry 2929 Consumer GoodsConsumer Goods

1515 Basic Materials ChemistryBasic Materials Chemistry 3030 Civil EngineeringCivil Engineering

0

5

10

15

20

25

30

TA3

TA2

TA1

FP6 Patents/ Technological ClassesFP6 Patents/ Technological Classes

SIC Codes: AlliancesSIC Codes: AlliancesClassClass descriptiondescription ClassClass descriptiondescription

13111311 Crude Petroleum and Natural GasCrude Petroleum and Natural Gas 48414841 Cable and Other Pay Television ServicesCable and Other Pay Television Services

28212821 Plastics Materials, Synthetic Resins, and Plastics Materials, Synthetic Resins, and Nonvulcanizable ElastomersNonvulcanizable Elastomers 48994899 Communications Services, Not Elsewhere Communications Services, Not Elsewhere

ClassifiedClassified

28362836 Biological Products, Except Diagnostic Biological Products, Except Diagnostic SubstancesSubstances 49114911 Electric ServicesElectric Services

35773577 Computer Peripheral Equipment, Not Computer Peripheral Equipment, Not Elsewhere ClassifiedElsewhere Classified 50455045 Computers and Computer Peripheral Computers and Computer Peripheral

Equipment and SoftwareEquipment and Software

35713571 Electronic ComputersElectronic Computers 50655065 Electronic Parts and Equipment, Not Electronic Parts and Equipment, Not Elsewhere ClassifiedElsewhere Classified

36513651 Household Audio and Video EquipmentHousehold Audio and Video Equipment 67946794 Patent Owners and LessorsPatent Owners and Lessors

36613661 Telephone and Telegraph ApparatusTelephone and Telegraph Apparatus 73727372 Prepackaged SoftwarePrepackaged Software

36633663 Radio and Television Broadcasting and Radio and Television Broadcasting and Communications EquipmentCommunications Equipment 73757375 Information Retrieval ServicesInformation Retrieval Services

36743674 Semiconductors and Related DevicesSemiconductors and Related Devices 73737373 Computer Integrated Systems DesignComputer Integrated Systems Design

36793679 Electronic Components, Not Elsewhere Electronic Components, Not Elsewhere ClassifiedClassified 73717371 Computer Programming ServicesComputer Programming Services

37143714 Motor Vehicle Parts and AccessoriesMotor Vehicle Parts and Accessories 73797379 Computer Related Services, Not Elsewhere Computer Related Services, Not Elsewhere ClassifiedClassified

37113711 Motor Vehicles and Passenger Car BodiesMotor Vehicles and Passenger Car Bodies 87318731 Commercial Physical and Biological Commercial Physical and Biological ResearchResearch

45124512 Air Transportation, ScheduledAir Transportation, Scheduled 87328732 Commercial Economic, Sociological, and Commercial Economic, Sociological, and Educational ResearchEducational Research

48124812 Radiotelephone CommunicationsRadiotelephone Communications 87428742 Management Consulting ServicesManagement Consulting Services

48134813 Telephone Communications, Except Telephone Communications, Except RadiotelephoneRadiotelephone

1311

2821

2836

3577

3571

3651

3661

3663

3674

3679

3714

3711

4512

4812

4813

4841

4899

4911

5045

5065

6794

7372

7375

7373

7371

7379

8731

8732

8742

0%

2%

4%

6%

8%

10%

12% TA1TA2TA3

FP6 Alliances / Relevant SIC

Cliques Vs. StarsCliques Vs. Stars

Assumption about the structure of the Assumption about the structure of the affiliation network: clique vs. staraffiliation network: clique vs. star

Assumption about the role played by Assumption about the role played by prime contractor:prime contractor:

no specific role for participants - no specific role for participants - clique hp.clique hp.

vs.vs.

A participant serves as the coordinating agentA participant serves as the coordinating agent - - star hp.star hp.

PC

PC

Clique vs. StarClique vs. Star Both assumptions: rather strong and somewhat Both assumptions: rather strong and somewhat

arbitraryarbitrary

Equally reasonable,Equally reasonable, they represent the upper they represent the upper and lower limitsand lower limits

So we explore the main topological So we explore the main topological

characteristics of the network under both characteristics of the network under both assumptionsassumptions

Indicative Analysis: 3 subjectsIndicative Analysis: 3 subjects

Subject 1: Identifying HUBs andSubject 1: Identifying HUBs and their relative roles their relative roles

Hub definitionHub definition

• An organization is a hub in a specific network if it has many links and/or if it connects the otherwise unconnected parts of the network

The above translates into high degree centrality and/or high betweeness centrality

STYLIZED 3A PARTNERSHIP STYLIZED 3A PARTNERSHIP NETWORKNETWORK

The node labelled “HUB 3a” is the designated Hub for this network.

This is a stylized model of Network

3a (Alliances)

Give intuition behind the concept

of a Partnership Hub

A Hub is defined as a node exhibiting high value of betweenness

and degree


Yellow nodes indicate

organizations participanting in

Framework Programme.


The yellow node represents the relevant Hub in the stylized 1a partnership network

The blue node is the 3a network relevant Hub

This is a stylized model of Network

1a (FP Participants)

Links Between 1a Hubs and 3a HubsLinks Between 1a Hubs and 3a HubsBlue nodes are the 3a network Hubs

Yellow nodes represent the

1a network Hubs

1a Hubs are strongly inter-connected and

they are also connected with 3a Hubs

3a Hubs are NOT hubs in network 1a, BUT are gateways that connect FP organizations to the global network

1A FP6 (TA1) PARTNERSHIP 1A FP6 (TA1) PARTNERSHIP NETWORKNETWORK

Blue nodes are the 3a network Hubs

Yellow nodes represent 1a

network Hubs

Red nodes are other 3a network

participants within distance 1 from 3a

Hubs

1A FP6 (TA1) PARTNERSHIP 1A FP6 (TA1) PARTNERSHIP NETWORK (no IP)NETWORK (no IP)

This is the TA1 Network without the

links related to IP

The network is substantially different,

with many isolated nodes and diminished

complexity

Subject 2: Effectiveness ofSubject 2: Effectiveness of KNOWLEDGE HUBs KNOWLEDGE HUBs

Effectiveness of Knowledge HubsEffectiveness of Knowledge Hubs

Hubs as knowledge depositories

Hubs at the cross-road of information and ideas

• Number of Patents • Number of Citations Received• Number of Highly Cited Patents

• Degree Centrality • Betweeness Centrality

Effectiveness of Knowledge Hubs: Effectiveness of Knowledge Hubs: Hypothetical ExampleHypothetical Example

patent citation/patent

betweenness IIIb

degree IIIb

highly cited patent

0

0,02

0,04

0,06

0,08

0,1

0,12

KH IIIa

KH Ia

OTHER Ia

Effectiveness of Knowledge Hubs:Effectiveness of Knowledge Hubs:Hypothetical ExampleHypothetical Example

• closely matches that of global KHs in terms of three variables closely matches that of global KHs in terms of three variables (number of patents, network centralities);(number of patents, network centralities);

• lags seriously behind in terms of the remaining two variables lags seriously behind in terms of the remaining two variables that approximate the quality and the importance of their patent that approximate the quality and the importance of their patent portfolios;portfolios;

the FP KHs seem to perform better in diffusing knowledge the FP KHs seem to perform better in diffusing knowledge through their centrality roles in the networks than in creating through their centrality roles in the networks than in creating powerful and influential portfolios of new ideas.powerful and influential portfolios of new ideas.

Subject 3: LeadershipSubject 3: Leadership

Leadership

Two different definitions of Leadership:

• Technology Leadership:Technology Leadership: the role played by each the role played by each organisation in the innovative process organisation in the innovative process

• Market leadership:Market leadership: the share of revenues in ICT the share of revenues in ICT among EU25 among EU25

Technology Leadership

Technology leadership is defined in terms of two Technology leadership is defined in terms of two concepts:concepts:

• Niche overlapNiche overlap concerns the crowdedness of the concerns the crowdedness of the technological area explored by organisations. Its technological area explored by organisations. Its measure is based on similarity of technological measure is based on similarity of technological antecedents (i.e. co-citation).antecedents (i.e. co-citation).

• PrestigePrestige deriving from the direct technological ties deriving from the direct technological ties between actors (i.e. direct patent citations)between actors (i.e. direct patent citations)


Four different kinds of actors:Four different kinds of actors:

• Technology Leaders:Technology Leaders: a key source of knowledge a key source of knowledge spillovers for many other organizations in the industry. spillovers for many other organizations in the industry. Their research activity is focused on the exploitation of Their research activity is focused on the exploitation of opportunities in relatively mature and therefore highly opportunities in relatively mature and therefore highly crowded fieldscrowded fields

• Technology Brokers:Technology Brokers: sources of knowledge in sources of knowledge in relatively new and relatively new and unexploredunexplored fields fields


• Technology Followers:Technology Followers: they they do not contribute do not contribute significant spillovers to other organizations and engage significant spillovers to other organizations and engage into relatively mature and crowded technological into relatively mature and crowded technological subfieldssubfields

• Isolate Organisations:Isolate Organisations: they they do not receive direct do not receive direct citations from many other organizations and are citations from many other organizations and are exploring relatively untapped technological subfields.exploring relatively untapped technological subfields.

Technology Leadership:Hypothetical Example

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

0 1 2 3 4 5 6 7 8 9 10

Crowding

Pre

stig

e

EU-Non FP

EU-FP KH

EU-FP Non KH

Global KH

Average prestige

Average alfa

Technology leaders

Technology isolates

Technology Leadership:Hypothetical Example

This analysis might suggest:

• The number of identified leaders and brokers that participate in the Framework Programme

• The number (and identity) of those who not only participate but they can also be characterized as Partnership HUBs in the Framework Programme.

Documents

Using Social Networks Methodology to Evaluate Research & Development Programs