March 11, 2003: I. Sim Research Informatics/Organizational IT Medical Informatics Ida Sim, MD, PhD March 11, 2003 Division of General Internal Medicine,

March 11, 2003: I. Sim Research Informatics/Organizational ITMedical Informatics

Ida Sim, MD, PhD

March 11, 2003

Division of General Internal Medicine, and Program in Biological and Medical Informatics

UCSF

Clinical Research Informatics/Organizational Aspects of Health

Information Technology Use

Copyright Ida Sim, 2003. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


Field Trip

• Palo Alto Medical Foundation- a state of the art EMR (EpicCare)– the promise and the reality

• Today Mar. 11– 1:30 to 3:30pm

• Directions sent out by e-mail


Outline

• Clinical research informatics– infrastructure for clinical research– systems for supporting clinical research

• Organizational aspects of IT use– examples of IT failures– reasons


Evidence Adaptive CDSSs

• CDSS recommendations should be evidence-based– should keep up-to-date with research findings– update mechanism should be semi-automatic

• Health care computing infrastructure should be integrated – for clinical care and decision support– for clinical research


Need For Informatics Infrastructure • “A nationwide effort is needed to build a technology-

based information infrastructure that would lead to the elimination of most handwritten clinical data within the next 10 years, the committee said. ...Without a national pledge to create and fund such a technological framework, progress to enhance quality of care will be painfully slow.” (IOM, Crossing the Quality Chasm, Mar 2001)

• IOM reports asks Congress to spend $1 billion on health informatics

• How do needs of clinical research and clinical care dovetail?


Joint Infrastructure for Care and Research

Administrative Clinical Care Research

ClinicalBilling

Physical Networking

Standard Communications Protocols (e.g., HL-7)

Standard Vocabulary

PracticeManagement

Systems

ElectronicMedicalRecord

??

Medical BusinessData Model

Clinical CareData Model

??


The Clinical Trial Cycle (per NCI)

New Ideas

Protocol & Funding

Findings

Approval &Preparation

Design Trial

Activate TrialConductTrial

UtilizeResults •trial simulators

•trial costing•protocol authoring

•IRB approval•CRF design

•data management•remote data entry•GCP compliance

•data analysis•reporting


Infrastructure for Clinical Trials

New Ideas

Protocol & Funding

Findings


Design Trial

Activate TrialConductTrial

UtilizeResults •trial simulators

•trial costing•protocol authoring

•IRB approval•CRF design

•data management•remote data entry•GCP compliance

•data analysis•reporting

• a few companies• many companies

• FDA electronic submission standards

• a few companies


Research and Care Together

for patientcare & policy

ClinicalCare

for clinical researchNew Ideas

Protocol & Funding

Findings


DesignStudy

ActivateStudy

ConductStudy

UtilizeResults for basic research


Joint Infrastructure for Care and Research

Administrative Clinical Care Research

ClinicalBilling

Physical Networking

Standard Communications Protocols (e.g., HL-7)

Standard Vocabulary

PracticeManagement

Systems

ElectronicMedicalRecord

Clinical Research Management

Systems

Medical BusinessData Model

Clinical CareData Model

Clinical StudyData Models


Case: Clinical Research Informatics• You are planning a study on infant jaundice...• What relevant studies have been completed on this

topic?• What ongoing studies should you know about?• You’re interested in running your study over the

web as much as possible.– what types of study activities can be done over the

web?– how good is the technology for these activities?


Relevant Trials: Completed

• Medline• Cochrane Controlled Trials Register

– ~328,000 records of controlled trials– manual logging of CCTs by hand searching journals– accessible from UCSF machines (IP address) only

• can set up proxy access

• metaRegister of Controlled Trials– 11,000 commercial and ongoing trials

http://www.updateusa.com/clib/CLIBINET.EXE?A=1&U=&P=

http://www.library.ucsf.edu/services/proxy/

http://www.controlled-trials.com/



Relevant Trials: Ongoing

• Non-profit/government– www.clinicaltrials.gov

• 5700 trials, ~3000 open• NIH-supported and some commercial cancer and AIDS trials

– cancertrials.nci.nih.gov– www.actis.org

• AIDS Clinical Trials Information Service – www.trialscentral.org (from Cochrane people)

• pointers to hundreds of clinical trial registries, by disease

http://www.clinicaltrials.gov/

http://www.actis.org/

http://cancertrials.nci.nih.gov/

http://www.trialscentral.org/


Relevant Trials: Ongoing

• Commercial: mostly for patient recruitment– www.centerwatch.com– www.ClinicalTrialFinder.com– www.controlled-trials.com– www.clinicaltrials.com– etc., etc., etc.

http://ww.centerwatch.com/

http://www.clinicaltrialfinder.com/


http://www.clinicaltrials.com/trials/trials.asp


Case: Clinical Research Informatics• You are planning on a study on infant jaundice...• What relevant studies have been completed on this

topic?• What ongoing studies should you know about?• You’re interested in running your study over the

web as much as possible.– what types of study activities can be done over the

web?– how good is the technology for these activities?


Clinical Study Tasks

• Project website• Subject recruitment• Eligibility determination• Protocol and forms distribution• Randomization• Data collection

– adverse events tracking


Industry is the Innovator

• RCTs now a $3.6 billion business (C. Scott, 7/00)

– in 1988, 95% of RCTs conducted by academics– now, over 80% conducted by industry

• Ergo, much of the technology innovation in clinical research execution is going on in industry– Applied Clinical Trials software directory

• http://www.actmagazine.com/appliedclinicaltrials/data/articlestandard/appliedclinicaltrials/452002/37090/article.pdf

• What’s the global picture for research informatics technologies?

http://www.actmagazine.com/appliedclinicaltrials/data/articlestandard/appliedclinicaltrials/452002/37090/article.pdf


Project Website

• Project website– GISSI website GISSI-x summaries, references

• http://wwwcardio.marionegri.it/trials.htm

– HERS main results revised tables from JAMA report• http://www.keeptrack.ucsf.edu/hers2/HERSfindat.htm

• Requirements– web server computer

• use a web hosting service (see http://www.cnet.com)• or have a web server program (e.g., Apache)

– pages of material• produce these using Word (save as HTML file)• or use a web editor (FrontPage, Dreamweaver)

http://www.irfmn.mnegri.it/other/cardio/gissi01.htm

http://www.keeptrack.ucsf.edu/hers2/HERSfindat.htm


JIFE Client/Server Model

itsa

jaundice

ucsf.edu

KaiserOakland

KaiserSanta Clara

Internet

at homedial-in to itsa.ucsf.edu via modem

pacbell.net

aol.com

LAN

KaiserSan Diego

• The “jaundice.ucsf.edu” computer hasweb server software. It “serves” web pagesin response to http commands such ashttp://jaundice.ucsf.edu/project-home.html


Project Website (cont.)

• Personnel– webmaster: handles the machine stuff– web designer: produces text & graphical content– database administrator/programmer: many databases

(e.g., FilemakerPro, Access, SQL) can now be exported to the web, but usually this involves moderate programming

• Status: easily doable today


Automated Eligibility Determination

• Study enrollment is big bottleneck• Eligible patients: patients whose characteristics

match with eligibility criteria• For computerized matching, need to have computer-

interpretable descriptions of– patient characteristics– the eligibility criteria

Match Eligible Patients

EMR

Eligibility Rule


Eligibility Example• Eligibility criterion: women who are 2 or fewer

years post-menopause, as defined in NCI’s Common Data Elements set

• Allowed values:Above categories not applicable AND Age < 50Above categories not applicable AND Age >=50Post (Prior bilateral ovariectomy, OR >12 mo since LMP with

no prior hysterectomy and not currently receiving therapy with LH-RH analogs [eg. Zolades])

Post (Prior bilateral ovariectomy, OR >12 mo since LMP with no prior hysterectomy)

Pre (<6 mo since LMP AND no prior bilateral ovariectomy, AND not on estrogen replacement)


EMR Data Needed

• Gender• Age• Time since LMP, whether

– 6 or fewer months, or 12 or more months

• Past surgical history– bilateral ovariectomy and/or hysterectomy

• Therapy– LH-RH analogs, or– estrogen replacement


Computer-Interpretable Eligibility Rule

• NCI working on common model for representing eligibility rules

• Logical rules– (Prior bilateral ovariectomy) OR (>12 mo since LMP AND no prior hysterectomy)– first order logic is the best representation model for this

• Temporal constraints – greater than 12 months since LMP...– representing time requires second-order logic

• Can do simple cases with database rules and triggers


Promising, but...

• Richly detailed EMR not widely available or well integrated

• Coding of eligibility rules is difficult• At present, can only expect computer to suggest

potential subjects. EMR can– prompt MD in real-time to refer patient to study, or– periodically batch notify MD of eligible patients, or– send letter of solicitation to patients

• Similar problems bedevil automated identification of guideline eligibility


Protocol and Forms Distribution• Allows for centralized forms management and storage

through a project website• If expecting users to download, print, fill out and fax form

back– need protocol and forms in electronic format (e.g.,Word or PDF)

• scan it using a scanner ($100-$4000)– makes an image of the page (e.g., .gif or .jpeg)

• optical character recognition (OCR) scanning– convert scanned text into an editable document (e.g., Word)

• Status: easily doable today


JIFE Forms Download

itsa

jaundice

ucsf.edu

KaiserOakland

KaiserSanta Clara

Internet


pacbell.net

aol.com

LAN

KaiserSan Diego

• “jaundice.ucsf.edu” “serves” forms such ashttp://jaundice.ucsf.edu/case-form.pdf for printing out


Protocol and Forms Distribution

• If expecting users to enter data online over the web– need someone to design the forms and build them to be

served over the web• e.g., using Access Visual Basic

– need security mechanisms (e.g., user login)– need data validation checks built into forms entry– data forms must send data to a database

• Status: doable with some programming


Infant Jaundice Online Forms

itsa

jaundice

ucsf.edu

KaiserOakland

KaiserSanta Clara

Internet


pacbell.net

aol.com

LAN

KaiserSan Diego

• “jaundice.ucsf.edu” “serves” online entry formssuch as http://jaundice.ucsf.edu/case-form.asp.Users enter data, which get checked at the clientside, and data is sent back to “jaundice.ucsf.edu.”


• Requirements– a web-based data collection form to collect patient information– programs to verify eligibility and randomize patient– program to generate a response to the enroller

– security, privacy, and backup provisions• Some commercial systems do this for you• Status: doable with some programming

Web-based Randomization

Project Central Enroller

patient info

randomization results


Electronic Data Capture• Fax• Voice

– most systems about 95-99% accurate for restricted domains

– can speak normally, but may need to train computer to your voice

• Handheld Electronic– your favorite PDA and docking station for downloading

to a database (over the web)– wireless PDAs can’t be far from a receiver node

• radiofrequency: slow, prone to interference• infrared: requires line of sight between PDA and node


Issues in Remote Data Capture

• Managing copies (local and central copies?)• Concurrent updates

– what if 2 people want to update same record?

• Merging data• Security and privacy• System downtime• System response time• Workflow issues


Adverse Events (AE) Monitoring

• Dedicated systems for capturing this data for FDA requirements– built in checks for adhering to GCP (Good Clinical

Practice)

• Standard electronic data model for adverse events pending– will simplify adverse events data exchange


Clinical Research Informatics Summary

• Project website– easily doable today

• Protocol and forms distribution– easily doable today

• Randomization– doable with some programming

• Data collection– fax– voice– electronic handheld– remote physiologic monitors

• Adverse events monitoring


Teaching Points

• Clinical care informatics infrastructure should be integrated with clinical research infrastructure

• Many aspects of clinical research can be done electronically, but in fragmented fashion

• Industry very active in clinical research– business imperatives dominate infrastructure design


Outline

• Clinical research informatics– infrastructure for clinical research– systems for supporting clinical research

• Organizational aspects of IT use– examples of IT failures– reasons


Learning Goals

• Develop a conceptual approach to analyzing and preventing IT implementation failures

• Appreciate importance of user-centered design, at all phases of an IT project


Failures: U Virginia

• 700-bed hospital, 200 staff, 1200 resident MDs, 3600 RNs• 1981: consultants recommended financial and clinical medical

information system (MIS)– projected savings $26.3 million over 5 years– payback time less than 2 years

• 1985: accounting, then AdmissionDischargeTransfer – accounts receivable decreased from 100 to 60 days

• 88-91: clinical functions added sequentially– dietary and radiology orders; lab orders, results retrieval; pharmacy

pathways;

• 92: full deployment in and outpatient– all orders, all lab results, xray reports in MIS

(Acad Med 68:20-25)


Failures: U Virginia (cont)

• Pros– pharmacy more efficient, fewer errors

• Cons– residents unplugged computers, refused to enter orders– 3 years behind schedule– 3x original cost estimate– Generated no reduction in staffing

• Governance– originally entirely by the Computing Services Group– final transition required special committee including of

Chairs of Medicine, Surgery, Peds


Failures: HP, MGH, etc. etc

• Hewlett-Packard EMR at Palo Alto VA– clinicians did not use the system

• display would log off after only a few minutes• huge monitors placed between doctor and patient

• MGH – results reporting system in 1970s led to resistance in

early 1990s to new systems


Classes of Failures• Total failure

– system never implemented or immediately abandoned

• Partial failure– major goals unattained, or significant undesirables

• Sustainability failure– lab ordering reduction systems

• Replication failure– “Bayesian” abdominal pain diagnosis system (DeDombal,

‘72)

• 8 study centers: diagnosis improved from 46% to 65% • laparotomy rate and appendiceal perforations fell 50%• not replicated elsewhere


Why Health IT (HIT) Fails

• Is it a shortcoming of technology?– didn’t capture the health problem correctly? – didn’t have enough info or was too slow?– used the wrong reasoning method? – insufficient diagnostic/modeling/ etc. performance?

• Was it poor interface design?• Was it lack of user training?• If a system fails, what can be learned to prevent

another failure?


General IT Management Challenges

• HIT failures are instances of general IT failures– California DMV spent $45m from 87-93 on aborted

electronic license system– IRS, INS, FDA, …$165m on CONFIRM car rental

and hotel reservation system (AA, Marriott, Budget, Hilton


What’s Common to IT Projects?

• Technology driven– a hammer looking for a problem

• e.g. “Wow I have a Palm Pilot! What can I do with this?”

• Problem driven– a problem looking for a solution

Initial ProblematicState of the World

ImprovedState of the World

Solution involving IT

InaccurateHeart Attack Dx

ImprovedHeart Attack Dx

Baxt’s Neural Network


Improvement Through Process Change

InitialState

Conceptualizationof Desired Change

ChangeProcess

AlteredState

Impetusfor change

(JAMIA 4:79-93)

OrganizationalBehavior &

ManagementPsychology &

Cognitive Sciences SociologySocial Psychology

& Social Anthropology

People and Organizational Issues


Conception–Reality Gaps• Conception–reality gaps often arise from mismatches

between formal and behavorial reality– formal: what’s official– behavorial: the real skinny

• Types of Gaps– technical, managerial, medical

• Dimensions of Gaps– Information– Technology– Processes– Objectives and Values– Staffing and Skills– Management and Structures– Other resources: money and time


Technical Rationality Gaps• aka the Org Chart Illusion• U Virginia order entry system allowed only MDs

to write electronic orders – subinterns (4th year med students) could not enter

orders for later co-signing– in non-computer version, subI writes orders and

resident later co-signs it • care is delivered, subI learns autonomy and decision-making• saves time for resident• returns time saved in the form of teaching (usually)

• Similar gaps for nursing care plans, worker time sheets, etc.


Managerial Rationality Gap• Managerial perspective

– increases in pharmacy expenses are the biggest component of escalating health care costs

– reduce drug costs (eg proton pump inhibitors) by negotiating specific price reductions (e.g., for lansoprazole)

– switch all patients to cheapest drug of class unless physician objects (from omep- to lanzoprazole)

• MD perspective– MD knows dosing, typical response, side effect profile of

omeprazole; managers provide no support for process change

• Patient perspective– patient is used to omeprazole, it works fine without side

effects, he doesn’t want to deal with a new medication


Medical Rationality Gap• Normative decision-making: e.g., mammography

in women aged 40-50

NoMammo

MammoBreast CA

No Breast CA

Breast CA

No Breast CA

FP

TNFN

TP

•Screen 10,000 women q 2 yrs to save one life•About 5,000 women will have a false positive


Conception–Reality Gaps• Conception–reality gaps often arise from mismatches

between formal and behavorial reality– formal: what’s official– behavorial: the real skinny

• Types of Gaps– technical, managerial, medical

• Dimensions of Gaps– Information– Technology– Processes– Objectives and Values– Staffing and Skills– Management and Structures– Other resources: money and time


Example CDSS• Diabetic retinopathy leads to many cases of

preventable blindness• Referral and followup rates for diab. ret. low• Tele-opthalmology for routine opthal screening

– primary care doctors have tele-eye stations (EyePACS) in their offices

– take digital images of patient retina– transmit images to opthalmologist for decision

on need for referral


ITP...

• Information – is system satisfying a true need of intended users?

• are retinal diseases not picked up? preventable blindness?

• Technology– is technical infrastructure available and realistic?

• broadband access to physician offices for teleopthalmology?

• Process– is new workflow significantly different from old?

• how to integrate into primary care visit workflow?


...OSMO

• Objectives and values– is it role/responsibility of internist to screen for eye

diseases?

• Staffing and skills– internist office personnel will run the tele-eye station?

• Management and structure– who owns the tele-eye station? who owns the retinal

images?

• Other resources– opthalmologists lose the business of routine eye exams


Summary of Org Change

• IT projects often fail (spectacularly)• Important source of failure is “people and

organizational issues”• Technology is a just a tool for process change

– can be positive and/or negative

• Better to consider people and organizational pitfalls during design and implementation– be careful if your research protocol involves new IT

Documents

March 11, 2003: I. Sim Research Informatics/Organizational IT Medical Informatics Ida Sim, MD, PhD March 11, 2003 Division of General Internal Medicine,