1

Exploring the Quality of Health Apps: The ISYS Ranking System

Immaculada Grau*, Fundación iSYS, Spain, Group for Research in Primary Care–IDIBAPS, Barcelona, SpainJosé Antonio Gallego Fundación iSYS, Spain, Francisco J Grajales III, University of British Columbia, Vancouver, CanadaLuis Fernandez-Luque*, Northern Research Institute, Tromso, NorwayBelchin Kostov, Group for Research in Primary Care–IDIBAPS, Barcelona, Spain

iSYS is a Foundation to develop social health projects in Internet

Medicine 2.0 Malaga, October 2014

2

Why the need for a scoring system in health apps?

• The number of mobile Apps on offer is huge, and is growing exponentially

• Users generally don’t know what they will find until they download an app and are faced with an overwhelming number of options.

• The trustworthiness of internet-based health information aimed at non-professional users is an important and widely debated topic.

Introduction

What are the problems…

What do we need…

• Users need security

• Scientific guarantee

• Empowerment through relevant information

4

Objectives

The main objective of the iSYS Ranking is to empirically evaluate the quality of health-related apps for patients within three domains:

• (A) Popular interest• (B) Quality (trust)• (C) Utility With this we aim to

1. Promote awareness of different criteria that will be helpful before downloading a health app.

2. Develop a public ranking of scoring to help developers improve the quality of their applications and promote excellence.

5

Objectives

• This method is not a quality or a safety seal. It is only a guide• Apps to evaluate would be for the general public –not for

healthcare professionals-.• The Apps evaluated are NOT part of a care process

(diagnosis and monitoring of clinical parameters ARE part of a care process), unless they were certified by a competent official body, like the FDA.

• The score is based mainly on the App description provided by the developer, and the first results of Google search.

Limitations:

6

Methodology

Inspired by: Moher D, Schulz KF, Simera I, Altman DG (2010)

Guidance for Developers of Health Research Reporting Guidelines. PLoS Med 7(2): e1000217. doi:10.1371/journal.pmed.1000217

1. Needs Assessment2. Literature Review3. Obtain Funding for the Rating (Guideline) Initiative4. Expert Sampling & Delphi Method5. Present and Discuss Results & prepare face-to-face meeting6. Develop the Rating System Draft7. Pilot Testing & Repeat steps 5 & 68. Develop an Explanatory Document 9. Publication and Knowledge Translation Strategy

7

Methodology

Literature Review

A broad search string was used to query articles published on the PubMed database in English and Spanish from 1993 to 2014, without geographic filters. Results:

• EVALUATION of health applications for Smartphone (23 results)• ANALYSIS of health applications for Smartphone (18 results)• QUALITY of health applications for Smartphone (17 results) A total of 18 articles met our inclusion criteria and their relevant data was extracted using a standardized template

Bender JL, Yue RYK, To MJ, Deacken L, Jadad AR. A Lot of Action, But Not in the Right Direction: Systematic Review and Content Analysis of Smartphone Applications for the Prevention, Detection and Management of Cancer J Med Internet Res 2013;15(12):e287

Outstanding

8

4. Identify Participants & Conduct a Delphi Exercise.

Methodology

Research Team (n=5) Technical team (n=4) Expert panel (n=16)

• Bibliographic review• Initial recommendations• Selection of external

Panel of experts

• Prepare 1st Questionnaire (Q1)

• Send out Q1

• Analysis and evaluation by the panel

• Prepare and send out 2nd Questionnaire (Q2)

• Analysis, statistics and evaluation by the panel

• Results report

• Specific evaluation for each recommendation

• Extra comments

• Analysis of results • Possible modification

and forwarding

1st delphi round

2nd delphi round

• Proof of Concept• 5 Apps

• 4 Patients• 3 MD• 3 Nurses• 2 Heath experts• 1 COM expert• 3 Tech experts

9

Methodology

On completion of the Delphi method, we carried out an assessment of five mobile apps. For this evaluation, items and mean scores obtained from Delphi were used.

Problem

(C) Utility scoring criteria did not work in any case, the research team decided to take on new criteria, based the literature on review.

These new criteria were initially given the lowest score (3 points on a scale of 1-5, where 5 was the maximum) obtained during the Delphi process, pending a new iteration from the panel of experts.

Solution

Both experts scored the five applications in turn and the score worked.

Limitation: These new criteria have not been evaluated by the expert panel, so the method will be released this year in a beta version.

10

Results

A. Popular Interest:

1. Positive Rating given by users who downloaded the App (4 points).

2. Available on 2 or more platforms (3 points)

3. Interest Statement by Patient/Affected Association (4 points).

SOURCE

• 11 criteria from Delphi method• 5 criteria from the new Literature review

11

Results

B. Trust / Quality 4. seal from a Health Agency maximum points (18 points) or

5. Validate by a specialized professional / Public Health Body / Scientific Society (4 points)

6. Promoted by Affected Association (3 points)

7. The App has a website (responsibility) / Data protection endorsement (4 points)

8. Cites sources of scientific evidence (4 Points)

9. Names the organization responsible (3 points)

12

Results

C. Utility 10. Wide research group RCT scores maximum (18 points) or

11. Research: No-RCT Small Group [<30 users] (3 points) 12. Declaration of professional scientific society or association / Partnership Declaration AFFECTED (3 points)

13. Provides information (3 points)

14. Provides info to useful Health Provider / trackers (3 points)

15. Links to other affected / users (3 points)

16. Uses games to promote health data (3 points)

13

Results: example

14

Results: example

15

Conclusions

1. The iSYS method was successful in scoring Health Apps

2. It compliments but does not substitute quality and safety seals.

3. It’s provides a neutral reference for a patient before they download an App.

16

Discussion

• We have not found any other objective valuation methodology to compare Apps.

• We believe this scoring method can make an active contribution to:• Empowerment of patients• Raise awareness of the importance of quality apps• Encourage excellence in the development of Apps• Contribute to measurement and development of

standards and guidelines

17

Next steps

• Develop a Web Site.• Find sponsorship• Seek and deal with social Feedback and

Criticism (crowd sourcing) to improve upon. • Update the Method. Periodic release of new

improved versions (Annual?)• Encourage Endorsement. An annual event to

reward best rated apps, developers and promoters

• Translate and publish the method.