Adoption and Value of Multimedia Radiology Reports ... · The PACS I refer to in this presentation...
24
Adoption and Value of Multimedia Radiology Reports Assessed by Hyperlink Usage, Click Through Rates and Analytics Presenting Author: Les Folio, DO, MPH, MSc, MAS Col (ret) USAF Lead CT Radiologist, NIH CC Adjunct Clinical Professor of Radiology, George Washington University Hospital Authors: Folio LR, Cohen G, Machado LB Radiology and Imaging Sciences, Clinical Center, National Institutes of Health
Adoption and Value of Multimedia Radiology Reports ... · The PACS I refer to in this presentation is the NIH Clinical Center PACS vendor \爀屲Carestream Health, that I also have
Adoption and Value of Multimedia Radiology Reports Assessed by Hyperlink Usage,
Click Through Rates and Analytics
Presenting Author: Les Folio, DO, MPH, MSc, MASCol (ret) USAF
Lead CT Radiologist, NIH CCAdjunct Clinical Professor of Radiology, George Washington University Hospital
Authors: Folio LR, Cohen G, Machado LBRadiology and Imaging Sciences, Clinical Center,
National Institutes of Health
Presenter
Presentation Notes
I am excited to share our experience with Multimedia-enhanced Radiologist Reports that include Hyperlinked Measurements, tables and graphs That we are finding are More Informative in general and more Consistent and Efficient in cancer trials.
s
Disclosures, Disclaimers, Conflicts of Interest
Research agreement with Carestream Health (Rochester, NY)(The NIH Clinical Center PACS shown in this presentation)
Government issued diagnostic imaging patents (no royalties)Author royalties (Springer)
This research was supported [in part] by the NIH Clinical Center Intramural Research Program
The content is the responsibility of the presenter and does not necessarily represent the official views of the National Institutes of Health
s
s
Objectives
• Demonstrate adoption rates of bookmarks and hyperlinks in MERR– among radiology subspecialties and modalities
• Analyze ordering provider's interaction with EMR and PACS– and report hyperlinks evidenced by click through analysis
• by referring medical subspecialties, institutes, research protocol
• Assess report value by exploring multimedia report interaction– by click through behaviors– objective evidence physicians are reading reports and content clicked
*EMR= Electronic Medical Record
Presenter
Presentation Notes
The PACS I refer to in this presentation is the NIH Clinical Center PACS vendor Carestream Health, that I also have a research agreement with where I will be sharing our combined efforts in developing advanced tools, … especially native reporting with dictation integrated PACS allowing for hyperlinking radiologist report text with image annotations producing….
s
Background, Development, Future• Radiology report has not changed dramatically over the years• Review survey radiologists & oncologists report preferences
– verified oncologists and radiologists prefer hyperlinks
• Annotations of radiologists findings in reports now hyperlinked– such as lesion measurements or ovals around lung pathology– unique opportunity to objectively analyze referring physicians CTR– can use these connections to train in supervised deep learning
s
NIH Clinical Center Oncologist &Radiologist Survey*
• Years ago we aimed to improve prior text-only radiology reports– Surveyed radiologists’ and oncologists’ reporting preferences – Tumor quantification was disjointed, inconsistent, tedious, inefficient
• Oncologists often measure lesions independently from report– Or search for measurements buried in our prior text only reports – It was tedious trying to match oncologist target lesions on images
• Survey verified oncologists and radiologists prefer hyperlinks– The report hyperlinks take clinicians to annotated measurements
* Folio L. Quantitative Radiology Reporting and Tumor Metrics: Survey of Oncologists and Radiologists. AJR. Oct 2015.
* Folio L. Quantitative Radiology Reporting and Tumor Metrics: Survey of Oncologists and Radiologists. AJR. Oct 2015.
Presenter
Presentation Notes
After all, our reports are the primary communication to referring clinicians; essentially “our product” However, they have not changed in over 100 years; since Roentgen To help improve radiologist report content, quantification and consistency with patients in cancer trials while improving efficiency, 109 oncologists and 28 radiologists survey replies at the NIH Clinical center verified oncologists make measurements on their own, often searching for measurements buried in our reports and hunting for metastatic lesions in PACS, consulting radiologists several hours a day, often only to make minor measurement tweaks and fill out forms with series, image number, measurement, lesion name resulting in duplicated efforts, discrepancies and severe inefficiency. Oncologists and radiologists agreed that hyperlinked reports would help start to answer some of these issues, with less discrepancies between radiology reports and records used to calculate response
s
Are NIH CC Radiology Reportsadequate for Oncology Assessment?
*Folio LR. Quantitative Radiology Reporting in Oncology:Survey of Oncologists and Radiologists. AJR. 2015
Presenter
Presentation Notes
To help improve radiologist report content, quantification and consistency with patients in cancer trials while improving efficiency, 109 oncologists and 28 radiologists survey replies at the NIH Clinical center Click verified oncologists make measurements on their own, often searching for measurements buried in our reports and hunting for metastatic lesions in PACS, consulting radiologists several hours a day, often only to make minor measurement tweaks and fill out forms with series, image number, measurement, lesion name resulting in duplicated efforts, discrepancies and severe inefficiency. Click Oncologists and radiologists agreed that hyperlinked reports would help start to answer some of these issues, with less discrepancies between radiology reports and records used to calculate response
Radiologists’ Satisfaction withCurrent Clinical History
Question:Current clinical history on imagingrequests is satisfactory forradiologists to provide tumor assessments.
Presenter
Presentation Notes
The NIH CC was an ideal place to perform the survey, with improving reports on patients in cancer trials that were especially deficient; with improvements having a high potential impact with oncologists that are especially demanding based on required consistent tumor measurements. It was not surprising to confirm that oncologists and even radiologists felt our reports were not sufficient for oncologists to assess tumor burden on their cancer patients in therapeutic trials. That is not unusual in the US as this is not an expectation in most centers since radiologists typically report on clinically applicable information. (CLICK) Unlike some countries where patients leave the radiology department with a RECIST determination. We aimed to improve our reporting and the entire process to exploit evolving technological advances. Our seemingly deficient reports were also in part due to….
s
0%
10%
20%
30%
40%
50%
60%
70%
80%
Report text (statusquo)
Reports withhyperlinked text
Tables Graphs Other
Oncologists
Radiologists
Question:How would you prefer tumor measurements presentation in radiology reports?
Presenter
Presentation Notes
inadequate clinical history on imaging requests, not unique to NIH Okay, … we don’t need a survey to reflect that radiologists are not satisfied with the current clinical history provided on imaging requests, but this lack of information up front begins the communication gap between oncologists and radiologists, leading to incomplete tumor assessments and supporting evidence that our prior process was broken. Incidentally, fewer histories include cancer type or baseline date and (CLICK) than this entertaining, however, inappropriately appropriate clinical indication, due to the ease of selection of this in menu dropdown selection I proposed drop down rules mandating cancer type and baseline date in our EMR, however, they may be trying to tell me something indirectly how they felt about my recommendations… (CLICK)
s
Interactive Report Components*
• Hyperlinked text to measurements, arrows, ovals, prior reports– Takes clinicians directly to the image slice in the CT or MR– Very helpful in CT and MRI exams with > 1,000 images, 20 sequences
• Hyperlinks easily created by dictating the word “hyperlink”• Bookmark1 tables of measurements (automatically generated)
• Additional multimedia elements (can be combined or separate)– Graphs representing lesion size changes over time on related2 lesions– Images, graphs or tables included in report or separate tumor report
* Folio L, Dwyer A, Machado L. Multimedia Enhanced Radiology Reporting: Concept, Components and Challenges. Radiographics. Mar 2018.
Presenter
Presentation Notes
This question about radiology report format is key in that a majority of radiologists and oncologists preferred more interactive reports (CLICK) with hyperlinked text, graphs and tables (CLICK) since we were co-developing this capability before implementation several years ago. Interestingly, the bookmark tables ended up being more important for data management than originally anticipated since the measurement data can be digitally exported to EMR and cancer databases to avoid handwriting on paper forms
s
FINDINGS:
Chest CT: Lungs, pleurae: lung nodules for example
right upper lobe (1.9 cm x 1.5 cm) (series 4, image 81)
who measured, when, relation and designation, name, lesion type
• “Active annotation” is either most recent measured, clicked or “b” shortcut
Record
Presenter
Presentation Notes
The main component that makes our reports interactive are the hyperlinked text to measurements or other annotations in the images These links take clinicians directly to the image within hundreds or thousands of cross sectional, ultrasound or IR fluoroscopy cine image data when clicked. We most commonly Link measurements, followed by prior reports in the comparison field, then arrows, ovals and text. Hyperlinks are easily created by dictating the word “hyperlink” importing the active annotation measurement, series and slice number while automatically generating bookmark lists in the background. Additional multimedia elements can include tables of measurements, graphs representing lesion size changes over time, or images, tables or graphs included in or linked to reports
This simulation demonstrates how the hyperlinked measurement, series and image number are imported with less cross checking between report and images. While dictating “Unchanged lung nodules right upper lobe” for example, (CLICK) Then the radiologist measures the lesion either manually or automatically, or already measured by a radiologist assistant, Then we simply say “hyperlink” (CLICK) to link to the active annotation, while automatically importing the measurement, image and slice number. significantly reducing cross checks with less errors. This is the start of the 7 or so data points that are included with each measurement, already in digital exportable format for oncologists. You will soon see how that becomes important…
s
… making it difficult for oncology staff to extract tumor measurements, slice and image number of target lesions…
Presenter
Presentation Notes
This is an example comparison of our prior text-only report where measurements were buried in inconsistent narrative text-only reports (CLICK)
s
Presenter
Presentation Notes
… making it difficult for oncology staff to extract tumor measurements, slice and image number of target lesions… Radiologists were often consulted to “drive” the PACS to guide oncology staff through the tedious hunting for measurements on reports, then finding the lesions and measurements within thousands of images, compounding the inefficiency.
s
FINDINGS:Chest CT:
Lungs, pleurae: Unchanged lung nodules for example right upper lobe (0.8 cm x 0.4 cm) (series 4, image 84)
Mediastinum, heart, great vessels: Unchanged mediastinal adenopathy for example subcarinal (2.5 cm x 1.4 cm) (series 2, image 27) and right hilar adenopathy for example (5.1 cm x 2.4 cm) (series 2, image 32) and (2.1 cm x 1.4 cm) (series 2, image 25)
GI Small and large bowel, mesentery, peritoneum: unremarkable
Pelvic CT: Central pelvis, sidewalls: Unchanged anterior pelvic wallmass.Osseous structures, spine, body wall, soft tissues: unremarkable
IMPRESSION: 1. Unchanged lung nodules 2. Stable mediastinal and hilar adenopathy/masses3. Unchanged anterior pelvic wall masses4. No evidence of new soft tissue mass
Multimedia Enhanced Radiology Reports
Presenter
Presentation Notes
However, (CLICK) now measurements and critical items stand out and interact with crucial imaging findings, mostly in our case, metastatic tumor measurements… where the hyperlinks take clinicians directly to the measured lesions that I will show on the next slide. Radiologists have also found this is easier since measurements, series and image number are imported directly into the report. The colored links also stand out to draw attention to them; also helping radiologists in finding lesions on the current exam that were previously measured.
s
Hyperlinked Text to Image Annotations• Hyperlinks to image findings seem to exemplify Clarke’s 3rd law1
“Any sufficiently advanced technology is indistinguishable from magic”– Thought provoking pre-implementation clinical staff presentations– Interactive reports result in fewer consultations to help “drive” PACS
• Help radiologists compare priors; jumping directly to key findings– Enhance readability of unstructured free text narrative reports– Links combined with standard RadLex templates compound efficiency
1 British science fiction writer Arthur C. Clarke2 Folio LR. Quantitative Radiology Reporting in Oncology: Survey of Oncologists and Radiologists. AJR. 2015
Presenter
Presentation Notes
This is my favorite slide demonstrating the interactive nature of the report where each link takes clinicians directly to (CLICK) the measurement 3D location (CLICK) within the CT series or MRI sequence measured, (CLICK) When the ordering provider or radiologist clicks the measurement, the co-registered exams both jump to the slice location, with the annotation highlighted in crosshairs; again click on this measurement… This results in a more interactive, easier to read report, with life; a monumental improvement over our prior text only reports. and with Radiologist assistants relating lesions over time, ..
We subdivided percent exams with bookmarks and percent of reports with hyperlinks by subspecialty.
94
76
45
13
26
86
6663
77
416
75
30
80
36
0 4
18
45
93
72
42
9
26
86
64
56
73
115
21 22
68
8 0 2
18
32
0
10
20
30
40
50
60
70
80
90
100
CAP
Ches
t
Abdo
men
and
Abd
+pel
vis
Neu
ro
MSK
Oth
er
Tota
l
Ches
t
Abdo
men
, pel
vis
Neu
ro
MSK
Oth
er
Tota
l
Ches
t, lu
ng, V
-P sc
an, T
orso
Body
, abd
omen
, pel
vis
Neu
ro
MSK
, ext
rem
ities
Oth
er
Tota
l
CTs MRI NM
Percentage of bookmarks and hyperlinks in May 2017 exams at the NIH
Bookmarks % Hyperlinks %
*Body CT:94% bookmarks93% hyperlinks
Highest due to need for consistent measurements
in cancer trials CT MRI NM
Gaskins C, Folio L. AI Driven Report Value. HIMSS Presentation. March 2018
Presenter
Presentation Notes
I believe we have exemplified Clarke’s 3rd law of technological advancements where he stated “Any sufficiently advanced technology is indistinguishable from magic” for example, When I fist presented how hyperlinks would guide to NCI oncologists years ago and at RSNA I first showed in 2015; the oohs and ahhs from the audience are very rewarding. I remember when our most popular ordering investigators first started using hyperlinks; their magic like amazement response was mostly realizing how the improved usefulness and efficiency would change their lives. The upcoming generation of medical trainees will expect this type of interaction as it will be on par with what they grew up with on their devices. (CLICK) a game changer for diagnostic radiology.
Physician Clicks from EMR to Thin PACS Viewer
Soon, report ”Click Through” analysis should provide a tool to confirm and assess the clinical value of hyperlinks
Institute
≈ 70%
Just over 70% of click through from our EMR to our PACS are from four NIH institutes.
Example action: Can help target EMR/ PACS training
Folio L, Dwyer A, Machado L. Multimedia Enhanced Radiology Reporting: Concept, Components and Challenges. Radiographics (schd Mar 2018)
UVA referred patient withmultimedia report to NIH
Presenter
Presentation Notes
This graph represents the number of accesses to our thin client PACS images and reports with links within our EMR showing that just over 70% of the exam click throughs to studies for May 2107 (CLICK) occur from providers from our top four referring institutes With the National Cancer Institute clicking the most (~34%) which again makes sense in that they are referring to our images and reports for the measurements for patients in cancer trials. We are further parsing out click analytics by ordering investigator, research protocol # and clinical specialty
Discussion• NIH CC is first in the world to implement; UVA 2nd also analyzing• Widespread adoption supports improved efficiency
• Radiologists do not adopt technologies or workflows that are not efficient
• Applying advanced hyperlink analytic tools to radiology reports• first objective evidence of report value• proving clinicians are reading reports; more so interacting with them
• UVA report CTR hyperlink analysis similar to NIH*• Hyperlinked text connected to image annotation will enhance AI
Gaskins C, Folio L. AI Driven Report Value. HIMSS Presentation. March 2018
Presenter
Presentation Notes
I get excited when we get outside structured and multimedia reports including hyperlinks and DICOM SR objects; injector data as in this case that seem to be on the rise. This exam was uploaded to our PACS from a UVA referred cancer patient with previous measurements readily visible thanks to annotations residing in the outside images and links standing out in reports from UVA. We are continuing to work on image exchange with vendors and outside centers. to make annotations, hyperlinks and bookmark lists/ measurement annotations compatible to minimize re measuring of outside exams, for example. This image set and some others I have shown will be included in a summary article on interactive multimedia in Radiographics scheduled for March 2018.
Hyperlink-to-Annotation to enhance Machine Learning
• Build on preliminary deep mining report/image data results1, 2
• Preliminary work demonstrates value of radiology reports as the basis of labeled training set for deep learning
• Lesion ID in phantoms with simulated interactive reports
• Radiologists text associated with annotations through links serve as the “supervised” deep learning labeling
• Enhanced semantic NLP reports provides ideal deep learning training set
1 Shin HC. Interleaved Text/Image Deep Mining on a Large-Scale Radiology Database for Automated Image Interpretation. 2016 J Machine Learning Research.2 Roberts K. A machine learning approach for identifying anatomical locations of actionable findings in radiology reports AMIA Annu Symp Proc. 2012.
Presenter
Presentation Notes
Emphasizing our radiology reports as the primary communication to referring clinicians, I believe it is imperative we take advantage of evolving technology to optimize report value; to all ordering providers, especially our oncologists at NIH that appear to have the most benefit and with evolving technology and sharing experience with workflows, including improving quantification consistency over time to determine therapeutic response
Deep Learning Hyperlink Annotation Schema
DA• Defined Architecture: cross sectional exams in PACS• Radiologist reports with hyperlinks to annotated images
SDL• NVIDIA 4x GPU on RHEL (run data locally, not cloud)• Apply Supervised Deep Learning (SDL) on labeled data
TM• Trained Model (TM) ready for test set of unknown CT exams
Test Set
• Apply test set of cross sectional exams with unlabeled pathology• Analyze results, eventually apply as decision support
Presenter
Presentation Notes
We are starting to build on existing work with interleaved hyperlink text deep mining supported by NLP now related directly to image findings in the images over time. Assisted by NLP in our workflow, now that we are labeling images with radiologists text as a form of supervised deep learning
Conclusions• Bookmark & hyperlink adoption by radiologists supports efficiency
• radiologists would not adopt inefficient workflows • we analyzed by radiology subspecialty, institute
• Body radiologists have highest hyperlink use at NIH• NCI clicked more links (than all institutes) in our EMR to PACS reports/ images• likely due to need for consistent quantification cancer trials
• Can assess report value objectively by referring clinician interaction• by analyzing CTR of hyperlinks within our EMR and hyperlinked reports • we verify physicians reading our reports; further interacting with them
• Hyperlinked text connected to image annotations will enhance AI• radiologists are the “supervision” of supervised deep learning with each report
We now have a unique opportunity to objectively analyze our report value with click through analysis and lastly, machine learning should be enhanced with hyperlinked text to image data as a form of supervised deep learning; Thank you for your attention….
References• Sadigh G, Hertweck T, Kao C, et al. (2015). Traditional Text-Only Versus
Multimedia-Enhanced Radiology Reporting: Referring Physicians’ Perceptions of Value. Journal of the American College of Radiology 12:5, 519-524.
• Folio LR, Nelson CJ, Benjamin M, et al. (2015). Quantitative Radiology Reporting in Oncology: Survey of Oncologists and Radiologists. AJR 205:3, W233-W243.
• Folio L, Yazdi A, Merchant M, Jones E. MultiMedia Tumor Reporting Appears to Improve Oncologist Efficiency in Assessing Tumor Burden: Initial Experience. RSNA Formal presentation 2015.
• Machado LB, Apolo AB, Steinberg SM, Folio LR. Multimedia-Enhanced Radiology Reports Improve Target Lesion Selection and Measurement Concordance, and Are More Efficient in Genitourinary Cancer Trials. Oral presentation. ARRS. April 19, 2016. Los Angeles, CA
• Gaskins C, Folio L. AI Driven Report Value. HIMSS Presentation. March 2018
Presenter
Presentation Notes
Here is my email if you have any questions or comments you would like to send during or after the meeting Also you can see our department and PACS in action on discovery channel if you have not seen already last summer. Thanks for your attention….
