April 23, 2008KT-KVM Workshop, NCDDR1 Knowledge Value Mapping Approach to Knowledge Translation: Its Relation to Evaluation and Management of R&D and Practical

April 23, 2008 KT-KVM Workshop, NCDDR 1

Knowledge Value Mapping Approach to Knowledge Translation: Its Relation to Evaluation and Management of R&D

and Practical Applications

Juan D. Rogers

School of Public Policy

Georgia Tech


Presentation Outline

General Problems of R&D Evaluation Use of Knowledge as Key Component in

R&D Performance Use-and-Transformation in Tech Transfer Examples from Knowledge Translation

Applications Developing Knowledge Value Maps


General Problems of R&D Evaluation

Political climate demands a “mechanical control” model Incompatible with well known traits of science:

More valuable results are the most unpredictable Every so often changes its mode of operation to suit new

challenges We have poor understanding of the socio-cognitive mechanisms

that cause many important things in the workings of research

Focus should broaden from “how” to “what” we are evaluating Almost always happens in actual R&D evaluation projects

Normative analysis: what values are realized


Knowledge Use and R&D Policy Performance

Outcomes, impacts, ultimate results happen in a different location from knowledge creation Social systems that must reflect impacts and outcomes are not the ones

that receive resources to do research

Many dissimilar communities and social actors are involved: E. Rogers - heterophily Complex knowledge flows are involved

Creates difficulties for R&D policy design and implementation and program management and evaluation Accountability is hard to articulate:

Due to the displaced outcomes vis a vis research activities Due to the poor understanding of the mediation processes

Effectiveness of transfer is inherent in R&D performance


Use-and-Transformation in Knowledge

Dynamics Knowledge Use and its cognates

Technology Transfer Diffusion of Innovations Knowledge transfer Knowledge translation

Beyond the “Linear Model” But how far?

Analytical frameworks reflect normative concern for impact rather than empirical knowledge of underlying phenomena

No pure use of knowledge in its flow Knowledge is transformed as it takes each step in its flow True of its process of creation More serious and less understood as it crosses community boundaries All relevant participants are hybrid user-producers of knowledge


Examples of Complexity of Knowledge Flows

Features of industrial networks affect change at the industry level (Andersson et al. 2006)

Knowledge flow from universities to industry: Some indicator uses encourage counterproductive strategies

(Langford et al. 2006). Knowledge flow in internal networks for effective

team work (Zhuge 2006). Enthusiasm about the potential of collaborations

Collaborative work can be very difficult to carry out in practice (Dustdar 2002).


A Knowledge Value Mapping Approach: Aims

KVM focuses on the knowledge flows between knowledge creation activities and the impact contexts

Management of R&D always includes concern for knowledge flow Internal to R&D process

Interdisciplinarity Portfolio management

External to R&D process Impact and outcomes Mission requirements Accountability to stakeholders

Knowledge flows are hard to manage May be facilitated to certain extent They are not uniform across relevant dimensions


A Knowledge Value Mapping Approach: Basis

Content-value structures affect knowledge flows Knowledge doesn’t have all its consequences in itself

Current evaluation takes only goals and objectives as relevant values When stakeholders are included, their values are taken as added or

conditional objectives (i.e. explicit interests) Process is substituted for value articulation

This perpetuates the fact-value dichotomy Values must be investigated empirically in the entire domain of

knowledge flow Interests and agendas matter, but Epistemic values such as validity and rigor criteria are generally not

analyzed for their effect on outcomes (intra-content) Values must be also pursued for different articulations


A Knowledge Value Mapping Approach: Procedure

Map the field Identify creators and users (we call them “Knowledge-Value

Collectives) Explicit references (publications, citations, etc.) Documented relationships (advisory roles, collaborations, consortia) Nominations

Find the patterned dynamics and formal agreements (organized actions, teams, collaborations, contracts, etc.; we call them “Knowledge-Value Alliances”)

Find the communication patterns Find evidence and infer the normative stances of all involved Articulate relevant patterns of knowledge flow


A Knowledge Value Mapping Approach: Applications

Retrospective KVM: Features of past or present knowledge flow Examples:

Division of knowledge labor in interdisciplinary teams or inter-sector teams

Emergence of new specialized roles in interdisciplinary research Institutional innovation in technology transfer Community valuation of hidden knowledge outcomes

Prospective KVM: Given KVM and proposed intervention, predict likely effects

Determination of facilitating and hindering factors for knowledge flows (relevant to evaluation and other policy concerns)


Application of KVM to Knowledge Translation in Health

Evidence of need for knowledge-value maps from the literature: Diversity of uses and contexts Insufficiency of research results for policy making Diversity of types and mechanisms of use of research results Variety of incentives in the context of use Multidimensional matching of interventions and contexts in AIDS and Substance

Abuse Complexities in a single field application Theoretical commitments in research Costs of knowledge absorption Professional identity of practitioners Values at different levels of change Contexts of practitioner decision making Interdisciplinarity Dissemination networks and effects on tech transfer


Underlying Value Issues in KT: Diversity of uses and contexts

Alignment with ideology facilitates adoption KT begins with assessment of research and

user’s preferences “hire a knowledge broker to help find and

develop those relationships” (Jacobson et al 2003)

Routine feedback of implementation lessons from the diversity of application contexts is key to effectiveness (Van Kerkhoff and Szlezak 2006)


Underlying Value Issues in KT: Insufficiency of research results

Complexity of the interface of research and policy making Incentives for both communities are necessary for

managing the interface (Lavis et al 2004; Lavis 2006) Introduction of new guidelines is costly and benefits

variable or ambiguous Research results are not sufficient guidance for policy

design and implementation Extra knowledge is not within health researchers’

competency to produce (Grimshaw et al 2004)


Underlying Value Issues in KT: Diversity of types and mechanisms of use of research results

There are three categories of uses: instrumental, conceptual and symbolic.

There are many explanatory mechanisms on the features of knowledge that facilitate its use: Engineering explanations Organizational interest explanations Two Communities explanations Interaction explanations

The category of use and the mechanism varies across agencies and policy domains. (Amara et al 2004)


Underlying Value Issues in KT: Incentives in the context of use

Implementation of evidence-based practice (EBP) in the field of occupational therapy (Welch and Dawson 2006): Practitioners in this field are embedded in a hierarchy

Most practice change is aimed at individual practitioners Numerous obstacles for this process to succeed:

perceived labor unfairness, misalignment of the EBP materials with their form of knowledge

acquisition and application in the field, lack of ownership of the content of EBP presented to them, lack of structures of mutual support for engaging in learning new

practice guidelines. “…learning framework that focuses on the intended users

of research as the most critical element of the process” (Farkas et al. 2003).


Underlying Value Issues in KT: Multidimensional matching in AIDS and Substance Abuse Cases

Organizational development methods to introduce EBP in substance abuse treatment (Amodeo et al 2006) Addresses multiple dimensions of change in the application context Does not address all dimensions of absorption

Substance abuse treatment programs (Bartholomew et al. 2007; Greener et al. 2007; Rowan-Szal et al. 2007) Higher organizational functioning led to better client engagement Also led to fewer barriers to training and greater staff satisfaction with

training Successful training programs were ones that received support from their

organizations to do post-training trials HIV prevention interventions (Eke et al 2006; Harshberger

et al 2006) Preparation during the research design to include contextual factors Intervention implementation requires ongoing proactive technical

assistance


Underlying Value Issues in KT: Complexities of a single field application

“Useful Field of View” technology for older drivers (Ball et al. 2003) move it to the field and application in a community setting.

What was involved: Establishing collaborations with health organizations in the area; Adaptation of the protocol to the field setting Training field personnel Finding consenting participants for field trials Multiple claims to ownership of the database Answering further questions of cost-benefits for the health care

system Legislation changes for purposes of changing required tests at

MVA


Underlying Value Issues in KT: Theoretical commitments in research

Theoretical shift in mental health research (“recovery vision”) has radical consequences for what counts as evidence (Anthony et al. 2003) Fundamental treatment outcome variables used in the EBM

approach become relatively unimportant Others related to patients’ experiences and goals as indicators

of recovery are more important Requires a commitment to develop evidence and explore its

implications The very content of EBM will be different depending on the path

researchers decide to follow in their theoretical commitment.


Underlying Value Issues In KT: More on epistemic values

Criteria that guide evidence assessment and quality change with field

EBM has its origin in two particular subfields of clinical medicine General internal medicine and clinical epidemiology Transferring evidence assessment criteria from them to other fields of

medicine creates problems (Upshur et al. 2001). EBM brings along the emphasis on quantitative measurement and,

especially, of randomly controlled trials (Worrall 2002). Outside of those fields these aspects of measurement and

evidence gathering may not be feasible or as important: “Although values are regarded as central to health-care decision making,

the quantitative orientation to evidence proposed by advocates makes normative discourse seem an afterthought.” (Upshur et al. 2001)


Underlying Value Issues in KT: The cost of knowledge absorption

Getting involved with new knowledge requires a significant investment of capabilities and resources: Concept of “absorption capacity”

Nature of efforts to enable the use of external knowledge Firms have their own research capabilities with the objective of

being able to absorb external knowledge (Cohen and Levinthal 1990).

Expectation of significant payoff for the commitment of resources to be made: scientific research will be applied if it is perceived to add value to

the user (Landry et al. 2006)


Underlying Value Issues in KT: Professional identity of practitioners

Values grounding the professional identity of nurses (Fagermoen 1997): “Professional identity is defined as the values and beliefs held by

the nurse that guide her/his thinking, actions and interactions with the patient.”

The actualization of other-oriented values is integral to their provision of nursing care They realize self-oriented values, such as the satisfaction and

reward in their work, through the realization of the other-oriented values.

Contextual factors limit their ability to provide care making their work less meaningful


Underlying Value Issues in KT: Values at different change levels

McGuire (1990) identifies 10 areas of difficulty for KT in nursing: complexity of change process; genesis of research programs; formulation of research questions; differences in theoretical approaches; timescales and planning cycles; information overload; credibility; applicability; response to change; management of change.

The target should not be at the individual level but at the organizational level: Critique of focus on behavior modification


Underlying Value Issues in KT: More on levels of change

Rushmer et al (2004) show lessons for changes in practices in primary care come from the lessons learned in organizational change and organizational learning.

Many of the obstacles faced by KT are typical organizational change issues: features of organizational leadership; commitment to the program; organizational culture; openness and trust; and so on.


Underlying Value Issues of KT: Contexts of practitioner decision making

Decisions to adopt clinical practices by physicians are made in complex networks of consultation and collaboration with colleagues (Hader et al. 2007) EBM guidelines expected to be followed independently Must trust the judgment of unknown experts Guidelines acquire meaning for the doctors within the entire

network of relations and exchanges Guidelines create questions about their ability to make critical

medical decisions Need reinforcement by trusted specialists Consistent with potential patient compliance Sustained support for the new practices

Not sufficient for doctors to produce new medical knowledge in the context of use.


Underlying Value Issues in KT: Interdisciplinarity

Empirical investigation in KT generally confined to the measurement of outcome variables related to the desired changes in behavior (Sudsawad 2007). KT has been defined normatively first Not the natural knowledge frontier of any particular field of

research. Numerous social science theories are relevant to KT

(Graham et al. 2006; Estabrooks et al. 2006). Education, diffusion of innovations, organizational behavior,

psychology, social communication, management, political science, among others

Attempts at reduction to first principles or a small set of dimensions, virtual new social science field of “KT research”


Underlying Values Issues in KT: Dissemination networks and effects on transfer of technology

variations

Pringle et al. (2004) studied 5 health dissemination networks in US (RADAR; CADCA; CSN; NTCP; The Network) Their ability to effectively disseminate federal-to-local information

depends on fine tuning in four normative dimensions: Synergistic specialization; Brokerage; Coordination; Broad access information systems

These networks are different from those in other content areas and other institutional environments

Reisinger et al. (2007) evaluated three alignment systems based on two different alignment theories for fabrication of mono-limb prostheses They had different performance outcomes, one theory faring

better than the other The different required equipment and processes led to different

paths of technology transfer, i.e. different ease of transfer


Concluding Remarks Knowledge flows are inherent in R&D

evaluation “Nobody’s land” between research and

impacts Must be determined empirically in the

evaluation process KVM is proposed to do so Content-value maps of user-producers

of knowledge

Documents

April 23, 2008KT-KVM Workshop, NCDDR1 Knowledge Value Mapping Approach to Knowledge Translation: Its Relation to Evaluation and Management of R&D and Practical