Software Engineering for Business Information Systems (sebis)
Department of Informatics
Technische Universität München, Germany
wwwmatthes.in.tum.de
Design of a Real-Time Web Application for
collaborative annotations in telemedicine.Master’s Thesis Final Presentation
Junus Ergin
1. Introduction & Scenarios
2. Research questions
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 2
Telemedical challenge
© sebis160429 Matthes Slides sebis 2016 3
Dentist Bob Radiologist John
Patient
1. Introduction & Scenarios
2. Research questions & Proceeding
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 4
Research questions
RQ1: What is the state of the art for real-time collaborative web based
annotation tools?
RQ2: What are the requirements for web based component design based on
the given scenarios in order to create real-time annotations over the internet?
RQ3: How can the tool be designed to be understandable,
and fulfill all functional requirements and quality attributes?
RQ4: How far does the implemented prototype satisfy the requirements
derived in RQ2 und how is an additional benefit guaranteed against other
typical solutions?
© sebis160429 Matthes Slides sebis 2016 5
Timeline
© sebis160429 Matthes Slides sebis 2014 6
15-05-15 16-11-15
Scenarios
Functional
RequirementsEvaluation
ImplementImplement
Quality
Attributes
Expert
survey
Mock-
up
design
Expert
validation
Architect
ure
State of the
Art
State of the Art
RQ1: What is the state of the art for real-time collaborative web based
annotation tools?
What strategies for synchronizing annotations exist?
How to lock critical sections, prevent mutual exclusion?
What frameworks for collaborative annotations exist?
How is complex content, as videos, annotated?
How to deal with different user types?
© sebis161115 Matthes Slides sebis 2016 7
Annotations
Lock critical sections Deal with different users Common solutions
1. Introduction & Scenarios
2. Research questions
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 8
Requirements
• 18 Functional Requirements
• Importance rated with survey
• Medical experts in teleconsultation were asked
© sebis161115 Matthes Slides sebis 2016 9
Synchronize in real-time Highlight parts in files
Turn annotation screen off Consistent color scheme
1. Introduction & Scenarios
2. Research questions
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 10
Mock-up design
2 different mock-ups based on survey
© sebis161115 Matthes Slides sebis 2016 11
All functions displayed Only important functions
Only menus
Everything
Only tools
Only frequent options
Mock-up 1
Mock-up 2
1. Introduction & Scenarios
2. Research questions
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 12
Implementation
© sebis161115 Matthes Slides sebis 2016 13
Scenarios
© sebis161115 Matthes Slides sebis 2016 14
Scenario
How to transmit data?
Implementation
© sebis161115 Matthes Slides sebis 2016 15
( )
Implementation
© sebis161115 Matthes Slides sebis 2016 16
Implementation – Live Demo
Live Demo
© sebis161115 Matthes Slides sebis 2016 17
1. Introduction & Scenarios
2. Research questions
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 18
Evaluation
• „Field test“ with real users
• Different Scenarios
• First usage was unaccustomed
• After 1-3 Minutes they understood everything
• They solved all tasked from the scenarios without experience
• Even people who had reservations before, solved all tasks
Rare functions confused the users
• Turning screen black
• Replay annotations
• Conformity with the survey results
© sebis161115 Matthes Slides sebis 2016 19
1. Introduction & Scenarios
2. Research questions
3. Requirements and their importance
4. Mock-up design
5. Architecture and implementation
6. Evaluation & Conclusion
7. Future work
Overview
© sebis161115 Matthes Slides sebis 2016 20
Future Work
• More research about understandable UI design
• Make functions more understandable
• Reuse the tool in phone apps
• Apply tool to other consultation fields
• Generic rendering in several file formats by manipulating the binaries
© sebis161115 Matthes Slides sebis 2016 21
Technische Universität München
Department of Informatics
Chair of Software Engineering for
Business Information Systems
Boltzmannstraße 3
85748 Garching bei München
Tel
wwwmatthes.in.tum.de
Junus Ergin
B.Sc.
+49 157 37033036
Thanks for your attention!
Proceeding
State of the art
© sebis161115 Matthes Slides sebis 2016 23
Scenarios
Functional Requirements Quality Attributes
Expert feedback and importance
Mockup design
Experts validation
Architecture Design and implementation
Field test + results
Introduction (Backup)
© sebis161115 Matthes Slides sebis 2016 24
Annotations
Support teleconsultation
with collaborative annotations
on different files
Scenarios (Backup)
© sebis161115 Matthes Slides sebis 2016 25
Scenario 1
Scenarios (Backup)
© sebis161115 Matthes Slides sebis 2016 26
Scenario 2
Scenarios (Backup)
© sebis161115 Matthes Slides sebis 2016 27
Scenario 3
Requirements (Backup)
© sebis161115 Matthes Slides sebis 2016 28
Highlight a part of a file Add text field to a file
Synchronize the view Draw a line on a file
Requirements (Backup)
© sebis161115 Matthes Slides sebis 2016 29
Turn screen off See other collaborators
Add shapes to a file Delete objects
Requirements (Backup)
© sebis161115 Matthes Slides sebis 2016 30
Support DICOM files Support video annotation
Support different image types Support PDF files
Requirements (Backup)
© sebis161115 Matthes Slides sebis 2016 31
Download annotated file Replay the annotations
Support files embedded via
pluginStore author information
Requirements (Backup)
© sebis161115 Matthes Slides sebis 2016 32
Consistent color scheme Display participant status
Quality Attributes (Backup)
1. Modularity / Reusability in all connected systems
2. Run within the browser environment
3. Deal with bad connection quality
4. Optimized data structure
5. Different screen sizes and devices
6. Robustness
7. Scalability
8. Don't store data on hard drive
9. Encrypted data transfer
10.End-to-end connection
11.Frontend in TypeScript / Angular 2
12.Backend in Python
13.Low effort of implementation
14.High Efficiency
15.Separation of concerns
© sebis161115 Matthes Slides sebis 2016 33
Why Typescript (Backup)
• All Javascript features
• + ES6
• + Types
• Very similar to Java
• Architecture much better
© sebis161115 Matthes Slides sebis 2016 34
Validation in backendDeal with conflicts
Validation in frontend
• Annotations can overlay
• Editing and deleting only possible
by one person
• How to prevent locks?
• -> Revert after a time Δ if there is
no update from the current editor
• Mutual exclusion?
• -> Not possible using Firebase
Implementation (Backup)
© sebis161115 Matthes Slides sebis 2016 35
annotation.currentEditor = 1
annotation.currentEditor = null
annotation.currentEditor = 2
NO! annotation.currentEditor = 1
Implementation Class Diagramm (Backup)
© sebis161115 Matthes Slides sebis 2016 36
Binary Semepahone (Backup)
© sebis161115 Matthes Slides sebis 2016 37
Source: http://flylib.com/books/en/1.410.1.108/1/
Implementation – Turn the arrow (Backup)
© sebis161115 Matthes Slides sebis 2016 38
Implementation – Show current state (Backup)
© sebis161115 Matthes Slides sebis 2016 39
Implementation – Draw line (Backup)
© sebis161115 Matthes Slides sebis 2016 40
Implementation – Dynamic Source (Backup)
© sebis161115 Matthes Slides sebis 2016 41
Implementation – File Service (Backup)
© sebis161115 Matthes Slides sebis 2016 42
Implementation – Annotation Service (Backup)
© sebis161115 Matthes Slides sebis 2016 43
Implementation – Consistent color (Backup)
© sebis161115 Matthes Slides sebis 2016 44
• Array with unique colors
• Doctor ID used in order to get a color
𝐶𝑜𝑙𝑜𝑟𝐴𝑟𝑟𝑎𝑦 𝑈𝑠𝑒𝑟𝐼𝐷 % 𝐶𝑜𝑙𝑜𝑟𝐴𝑟𝑟𝑎𝑦. 𝑙𝑒𝑛𝑔𝑡ℎ
UserId % length Hex Code
0 FF6E6E
1 EF2121
2 FFA26E
3 EF6B21
4 92A4D7
5 465DA2
6 57B7C4
7 198B9B
8 19B264
9 FDCBCC
10 C36567
Literature reseach keywords (Backup)
Sources were searched at mediaTUM
-> Includes all important indices, as Google Scholar and Deutsche Nationalbibliothek
• Annotations
• Annotation framework
• Collaboration conflicts
• Collaboration framework
• Collaboration internet
• Collaborative annotations
• Collaborative workspace
• Critical section data lock
• Critical section problem
• DICOM annotation
• Medical annotation
• Real-time collaboration
• Real-time lock
• Real-time synchronization
• Synchronize data
• Telemedicine annotation
• Telemedicine collaboration
• Telemedicine real-time
© sebis161115 Matthes Slides sebis 2016 45
State of the art sources (Backup)
[Mich 14] P. N. R. K. M. J. Michael Derntl, Stephan Erdtmann. Echtzeitmetamodellierung
im Web-Browser. RWTH Aachen, Chair of Informatics 5, 2014.
https://pdfs.semanticscholar.org/0829/cada43a4b57612366885763ebb555578a6f0.pdf
[Qi] G.-J. Qi, X.-S. Hua, Y. Rui, T. Mei, and H.-J. Zhang. \Correlative multi-label video
annotation". http://dl.acm.org/citation.cfm?id=1291245
[Kell 07] M. A. Kelley and M. S. Wengrovitz. Real-time collaboration center. 2007.
https://www.google.com/patents/US8737596
[Rajk 12] R. Rajkuma. Synchronization in real-time systems: a priority inheritance approach. 2012.
https://books.google.de/books?hl=delr=id=jYICAAAQBAJoi=fndpg=PP10dq=
real+time+synchronizationots=sqMyEPcGA1sig=
LMTMWXPXpLeYWoh1hMIcPsB3wG8v=onepageq=real
[Sha 02] L. Sha, R. Rajkumar, and J. Lehoczky. Priority inheritance protocols: an approach to
real-time synchronization. 2002. http://ieeexplore.ieee.org/document/57058/
Fox 15] Fox, Wu, Uyar, and Bulut. \A Web Services Framework for Collaboration and
Audio/Videoconferencing". 2015.
[Herl 93] M. Herlihy and J. E. B. Moss. Transactional memory: architectural support for lock-free
data structures. ISCA '93 Proceedings of the 20th annual international symposium on
computer architecture, Pages 289-300, 1993. http://dl.acm.org/citation.cfm?id=165164
[Teva 87] A. Tevanian. MACH threads and the UNIX kernel:
the battle for control. Carnegie Mellon University, 1987.
http://repository.cmu.edu/cgi/viewcontent.cgi?article=2728context=compsci
© sebis160429 Matthes Slides sebis 2016 46
State of the art sources 2 (Backup)
[Maek 85] M. Maekawa. An algorithm for mutual exclusion in decentralized
systems. ACM
Transactions on Computer Systems Volume 3 Issue 2, May 1985 Pages 145-159,
1985. http://dl.acm.org/citation.cfm?id=214445.
[Chen 05] T. Chen, R. Lai, and A. Chen. System and method for setting
user-right, and recording medium. US Patent 20050144060 A1, 2005.
https://www.google.com/patents/US20050144060
[Vond 11] C. Vondrick and D. Ramanan. Video annotation and tracking with active
learning. Advances in Neural Information Processing Systems 24 (NIPS 2011),
2011. http://web.mit.edu/vondrick/vatic/videoalearn.pdf
[Holz 14] M. Holzer. Konzeption und Realisierung eines Frameworks f•ur verteiltes
Rechnen in
Webbrowsern mittels WebRTC. Master's thesis, 2014.
[S 13] M. de Sa, D. A. Shamma, and E. F. Churchill. \Live mobile collaboration for
video production: design, guidelines, and requirements". 2013.
© sebis160429 Matthes Slides sebis 2016 47