Running head: EVALUATION OF APPOINTMENT REMINDER SYSTEM
1
Evaluation of a Reminder System for Perinatal Psychiatric Appointments
Lacy R. Clayton, PMHNP-BC, MSN, RN
Doctor of Nursing Practice
DNP Project Chair: Patti Rager Zuzelo
Faculty Advisor: Joanne Serembus
Drexel University
September, 2018
EVALUATION OF APPOINTMENT REMINDER SYSTEM
2
Ó Copyright 2018
Lacy Rae Clayton, All Rights Reserved
EVALUATION OF APPOINTMENT REMINDER SYSTEM
3
Table of Contents
Page
Abstract 7
Chapter One: Introduction and Overview of the Problem 8
Problem Statement 10
Current Practices 10
Purpose of the DNP Project 11
Project Objectives and Outcomes 11
Clinical Question (PICOT) 12
Chapter Two: Review of Literature/Synthesis of Literature 13
Search Table 12
Appraisal of Evidence 14
Limitations 21
Summary & Conclusion 22
Chapter Three: Conceptual Foundation or Model 23
Conceptual/Theoretical Model 24
Conclusion 25
Chapter Four: Project Design 25
Project Implementation Steps 25
Practice Setting 26
Key Informants 27
Practice Change/Quality Improvement Design 27
Data Collection Tools 27
EVALUATION OF APPOINTMENT REMINDER SYSTEM
4
Data Collection Methods 28
Institutional Review Board (IRB) Approval 28
Conclusion 29
Chapter Five: Implementation Processes/Procedures 29
Letters of Permission 30
Resources 30
Conclusion 30
Chapter Six: Evaluation and Outcomes of Project 31
Key Informants 31
Methods 31
Data Analysis 32
Results 32
Conclusion 35
Chapter Seven: The Essentials of Doctoral Education for Advanced Nursing Practice 36
Implications to Practice/Policy Changes 36
Essential I: Scientific Underpinnings 37
Applicability to Project 37
Implications for Practice 37
Essential II: Organizational and Systems Leadership 37
Applicability to Project 38
Implications for Practice 38
Essential III: Clinical Scholarship and Analytical Methods 38
EVALUATION OF APPOINTMENT REMINDER SYSTEM
5
Applicability to Project 39
Implications for Practice 39
Essential IV: Information Systems/Technology 39
Applicability to Project 39
Implications for Practice 40
Essential V: Healthcare Policy for Advocacy 40
Applicability to Project 40
Implications for Practice 40
Essential VI: Interprofessional Collaboration 41
Applicability to Project 41
Implications for Practice 41
Essential VII: Clinical Prevention and Population Health 42
Applicability to Project 42
Implications for Practice 42
Essential VIII: Advanced Nursing Practice 42
Applicability to Project 43
Implications for Practice 43
Chapter Eight: Project Implications and Conclusions 44
Project Strengths and Limitations 44
Significance/Implications 45
Conclusion 45
EVALUATION OF APPOINTMENT REMINDER SYSTEM
6
References 46
Appendices 54
EVALUATION OF APPOINTMENT REMINDER SYSTEM
7
Abstract
Background: Rates for missed appointments (no-shows) are especially high in psychiatry.
Perinatal women are particularly at risk for missing appointments due to logistics such as
childcare and transportation. Though rates of depression are higher for perinatal women than the
general population, perinatal women often remain untreated. Appointment reminder systems are
an effective method for increasing appointment attendance and cancellations. Literature suggests
providing two reminders within seven days of the appointment. Maintaining accurate patient
contact information is a challenge to delivering reminders.
Purpose: Compare the appointment reminder system of a large, urban hospital to evidence-based
practice in an effort to reduce the 60% no-show rate within its perinatal psychiatry service.
Methods: A two-hour work-group meeting with five identified key experts evaluated the
appointment reminder system within the perinatal psychiatry service. A semi-structured
interview compared the intended appointment reminder process to its actual delivery. Strengths
and challenges were identified.
Results: The institution’s appointment reminder system consists of one automated reminder 48
hours prior to the appointment. The scheduling database is not updated to reflect patients who
cancel appointments, causing available appoint times to remain unfilled. Depending on the
referral source, some patients receive additional personal reminders from clinicians.
Conclusions: The current appointment reminder system may benefit from adding an additional
reminder based on evidence-based practice which suggests two reminders are preferred.
Attention should be given to appointment cancelations to optimize provider time and increase
patient access.
Keywords: appointment reminders, SMS reminders, appointment no-show.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
8
Evaluation of a Reminder System for Perinatal Psychiatric Appointments
Chapter One: Introduction and Overview of the Problem
Inadequate treatment of mental-health diagnoses is prevalent in the United States. Nearly
50% of Americans experience depression, anxiety or another mental health diagnosis throughout
their lifetime (Andrews & Thomson, 2009). Depression is the leading cause of disability for
persons age 15-44 in the United States (National Institute of Mental Health, 2018). The National
Alliance on Mental Illness (NAMI) reports an estimated 60% of Americans with mental illness
remain untreated (2018). Minority groups and underserved populations are especially at risk for
undertreatment. Studies indicate that only 21% of minorities meeting criteria for major
depression receive treatment for their illness (Gonzalez et al., 2010; NAMI, 2018).
Perinatal women are particularly vulnerable as it relates to mental health. The frequency
of depression among postpartum women is higher than depression rates for the general
population, with one in seven women receiving a diagnosis of postpartum depression (American
Psychiatric Association, 2017) compared to one in ten women within the general population
(Centers for Disease Control, 2017). Yet the mental health needs of this group remain
underdiagnosed and undertreated (Geier, Hills, Gonzales, Tum & Finley, 2014).
Inadequate treatment of perinatal mental health has a substantial impact on both mother
and child. Untreated postpartum depression contributes to decreased mother and child bonding
(Moehler, Brunner, Wiebel, Reck & Resch, 2006). Poor maternal bonding significantly impacts
infant development and safety. Mothers with unmanaged mental illness are less likely to engage
in stimulating activities such as reading, playing, and affectionate touching. These mothers are
less attuned to their child’s cues regarding hunger or illness and may not pay proper attention to
safety matters such as child proofing or car seat safety (Field, 2010). Children whose mothers
EVALUATION OF APPOINTMENT REMINDER SYSTEM
9
have untreated depression are more likely to be diagnosed with depression and oppositional
defiant disorder during their adolescence (Byatt et al., 2012).
Weinreb et al., (2014) found that 35% of pregnant women with a documented diagnosis
of depression did not receive mental health treatment during their pregnancy. Similarly, Geier et
al. (2014) found only 48% of pregnant women with a diagnosis of depression received treatment
compared to 72% of non-pregnant women diagnosed with depression. The main predictor for
the duration of a postpartum depression episode is the amount of time lapsing before treatment is
initiated, making early detection and treatment crucial for these women and their families
(Hansotte, Payne & Babich, 2017).
Women requiring evaluation and treatment for postpartum depression receive necessary
services only when they present to providers; high no-show appointment rates represent a serious
underutilization of important mental health services with potentially far-reaching adverse effects
on maternal-child health outcomes. Research supports that perinatal women are particularly at
risk for missing scheduled appointments due to logistics such as transportation and childcare
(Grote et al., 2015).
Given the need for mental health treatment, it is important to ensure follow-through with
appointments once they are scheduled. Though missed appointments are experienced across all
medical specialties, evidence suggests that the rate of psychiatric missed appointments may be
twice that of other specialties (Mitchell & Selmes, 2007). Reported rates for psychiatric missed
appointments range from 15%-28% with rates for missing initial appointments typically being
higher than rates for follow-up appointments (Mitchell & Selmes, 2007).
The financial impact of missed appointments is substantial, with an average loss of $200
for each individual missed appointment within a healthcare agency and a total loss of 150 billion
EVALUATION OF APPOINTMENT REMINDER SYSTEM
10
dollars annually to the United States healthcare system (Kheirkhah, Feng, Travis, Tavakoli-
Tabasi, & Sharafkhaneh, 2016; Zimmerman, 2014). This is particularly relevant to psychiatry
considering untreated mental health diagnoses are said to cost 100 billion dollars annually in lost
productivity (Scientific American, 2012). Additionally, missed appointments are associated with
poorer health outcomes, including reduced rates of medication adherence (Mitchell & Selmes,
2007), increased symptoms, greater risk of hospitalization with longer stays and increased
hospital readmissions rates (Gajwani, 2014). Rates of missing initial appointments are higher
than non-attendance rates for follow-up appointments (Mitchell & Selmes, 2007).
Problem Statement
Missed appointments have a substantial impact on the financial well-being of the
healthcare system and on the physical and emotional well-being of patients not receiving their
scheduled care. Appointment absenteeism is of particular concern in a psychiatric setting given
this specialty’s high missed appointment rate and the impact of untreated mental health on
patients and society. Although evidence exists on best practices for appointment reminder
systems to improve appointment adherence, these recommendations are not universally
implemented across healthcare settings (McLean et al., 2014). The Department of Obstetrics and
Gynecology at a large teaching hospital in Philadelphia, PA currently has a 35% overall no-show
rate for their obstetrics practice. This rate increases to 60% for missed appointments within the
perinatal psychiatric service from January to August, 2018.
Current Practice
Appointment reminder systems are common practice for reducing the amount of missed
appointments. Reminder systems such as mailings, telephone calls, and text messaging all show
some degree of improving appointment adherence (Finkelstein, Liu, Jani, Rosenthal, &
EVALUATION OF APPOINTMENT REMINDER SYSTEM
11
Poghosyan, 2013). Research findings support that tailoring the reminder system based on patient
preferences and utilizing multiple modalities results in higher rates of appointment adherence
(Crutchfied & Kistler, 2017).
While developing this DNP project, it was unknown if the institution’s appointment
reminder system adhered to evidence-based best practices or what challenges were experienced
related to the system. This prompted an evaluation of the reminder to gain insight on potential
improvements which could thereby increase appointment attendance and patient access to care.
Purpose of the DNP Project
The purpose of this Doctor of Nursing Practice (DNP) evidence-based project was to
complete an evaluation of the appointment reminder system for perinatal psychiatric
appointments within a large urban hospital’s outpatient obstetrics clinic. The evaluation was
comprised of two steps. First, the design of the current reminder system was evaluated
compared to the current literature regarding best practices for appointment reminder systems.
Second, process mapping based on a key informant work group was completed to provide a
detailed understanding of how the reminder system is actually delivered and what challenges are
experienced. These two steps identified gaps between the current program design and evidence-
based practice and identified gaps between the intended program design and the delivered
program.
Project Objectives and Outcomes
The main objective of this DNP project was the completion of a comprehensive process
evaluation of the appointment reminder system for perinatal psychiatric appointments scheduled
within this obstetrics clinic. This evaluation included acquiring an understanding of how the
reminder system is intended to be implemented and determining how this compares to the
EVALUATION OF APPOINTMENT REMINDER SYSTEM
12
delivered reminder system. This information was gathered through a one-time work group
meeting with key informants. The delivered system is represented by a flow chart/process map
diagram constructed in consultation with key informants to provide a detailed, accurate pictorial
representation of the delivered appointment reminder program. The process map was compared
to current evidence-based practices for optimizing appointment adherence. Recommendations
were solicited from key informants. The primary outcome was an Executive Summary of main
findings and recommendations. The Executive Summary was distributed to key administrative
stakeholders within the institution with the purpose of offering evidence-based solutions for
revising the appointment reminder system with the intent of increasing appointment adherence.
Clinical Question/PICOT
This DNP project evaluated the current appointment reminder system of a perinatal
psychiatric service within an outpatient obstetrics clinic located in a large urban hospital in
Philadelphia. Psychiatry is the clinic’s most requested referral and it currently offers co-located
psychiatry appointments by a psychiatric nurse practitioner one day per week. The psychiatry
service has a no-show rate as high as 61% despite the requested demand. This program
evaluation offers a comprehensive understanding of the current appointment reminder system,
comparing it to the existing literature regarding best practices for appointment reminder systems
and soliciting input from key experts. The clinical question to be answered is, how does the
program’s current appointment reminder system compare to evidence-based practices for
optimizing appointment adherence? This question was constructed using the PICOT format:
Population: Perinatal women scheduled for psychiatry appointments; Intervention: Evidence-
based evaluation of current reminder system; Control: Current reminder system; Outcome:
Improved appointment attendance rates; and Timeline: Four weeks.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
13
Chapter Two: Review of the Literature
A comprehensive literature review examining best practices for appointment reminders
was conducted. PubMed, CINAHL, and Cochrane Library databases were searched. Key search
words included: appointment reminder systems, text/SMS reminder system, no-show,
appointment adherence, and improving appointment attendance. Specific information regarding
key search terms and number of retrievals by database can be found in Table 1.
Search Results
Table 1A PubMed Search Terms and Search Results Search terms Number of hits Improving appointment attendance 85 Increasing appointment attendance 42 Appointment reminder 373 Text AND appointment reminder 151 SMS AND appointment reminder 70 Mental health AND no-show 29 Search range 2007-2018 Table 1B CINAHL Search Terms and Search Results Search terms Number of hits SMS reminder 25 Text reminder 14 Appointment attendance 58 Appointment adherence 30 Appointment reminder 87 Text AND appointment AND reminder 70 Mental health AND no-show 15 Search range 2007-2018 Table 1C Cochrane Library Search Terms and Search Results Search terms Number of hits
EVALUATION OF APPOINTMENT REMINDER SYSTEM
14
SMS reminder 6 Text reminder 15 Appointment reminder 10 Search range: All
Studies researching telephone and/or text appointment reminder systems to promote
appointment attendance were reviewed. A total of 151 abstracts were returned based on key
search terms. These articles were reviewed based on established inclusion and exclusion criteria.
Eligible articles were published within the past 10 years. To maintain cultural and language
similarities, studies were from North America, The United Kingdom, and Australia. Eligible
studies focused specifically on routine appointment reminders. Studies of reminder systems for
promoting medication adherence or prompting patients to adhere to highly regimented
appointment schedules such as HIV treatment or dialysis were excluded. Studies focusing on
rural areas were also excluded considering this project took place in an urban setting. Studies
were critically appraised for quality and research design and preference was given to randomized
controlled trials and experimental designs. After assessing abstracts for duplicates, design
quality, and meeting inclusion criteria, 16 articles were included for review.
Appraisal of Evidence
The effectiveness of appointment reminder systems has been the subject of an extensive
body of research. Based on the established inclusion and exclusion criteria, 16 articles were
identified as the most relevant to support and guide this DNP project. Five of the studies were
systematic reviews or meta-analyses. Four randomized controlled trials (RCT) and one
prospective, randomized, parallel design trial were included. One study was a quasi-
experimental design. The final five studies consisted of Level III or IV observational cross-
EVALUATION OF APPOINTMENT REMINDER SYSTEM
15
sectional, case controlled or convenience sample designs. Refer to the Table of Evidence
(Appendix A) for additional information.
It is established that appointment reminders are effective. A synthesis of 31 RCTs and 11
systematic reviews found that reminders consistently improve appointment attendance (McLean
et al., 2014). However, there is no current definitive best practice guideline for appointment
reminders. In their systematic review, Gurol-Urganci et al. (2013) found that telephone and text
message reminders were equally effective in improving appointment adherence. This literature
review identifies best practices and subsequently evaluates how these practices compare to usual
care within the institution.
Systematic reviews. The five systematic reviews had overarching goals as well as their
own unique perspectives. The number of studies included in each review ranged from eight to
sixty. There was duplication of studies included across the systematic reviews. Gurol-Urganci,
de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) only included RCTs in their Cochrane
Review (N=8), while the other four reviews included a combination of RCTs and other study
designs in their analysis. Three of the reviews focused on digital or text message reminders
while two reviews combined telephone and text message reminders. Each of the reviews
concluded that reminder systems were effective tools for improving appointment adherence.
Telephone reminders. Automated or live telephone calls are perhaps the most common
modality for reminder systems and they are consistently found to be effective. The question of
how many and what format is an important consideration for optimizing a reminder system.
Shah et al. (2016) conducted an RCT to study the impact of a personal reminder call made seven
days prior to the patient appointment compared to usual care of an automated reminder three
days prior to the appointment. The intervention arm had a significantly lower no-show rate than
EVALUATION OF APPOINTMENT REMINDER SYSTEM
16
usual care, 22.8% compared to 29.2%. The intervention arm was also more likely to cancel
their appointment ahead of time, allowing opportunity for another patient to be scheduled. Teo,
Forsberg, Marsh, Saha and Dobscha (2017) studied a convenience sample of patients (N = 250)
at the Department of Veteran’s Affairs. They found patients receiving live reminder calls were
significantly more likely to attend appointments than patients receiving a reminder message or
no reminder, with no-show rates of 3%, 24% and 39% respectively. Parikh et al., (2010)
performed a prospective, randomized, parallel design clinical trial comparing three appointment
reminder groups – personalized telephone reminder, automated telephone reminder, and no
telephone reminder. The study confirmed prior findings that no-show rates are significantly
higher for initial appointments compared to follow-up appointments. No-show rates were
significantly lower in the personalized (13.6%) and automated (17.3%) intervention compared to
no intervention (23.1%). Cancellation rates were also significantly higher in the two intervention
groups compared to the control group.
These studies support the effectiveness of telephone reminders for increasing
appointment attendance. Though telephone reminders are effective in general, live or
personalized reminders are more effective than automated reminders. The two RCTs had sample
sizes of N=2,000 or greater and used a 95% confidence interval (CI) to determine differences
between the groups. Teo et al., (2017) studied a more homogenous sample within the
Department of Veteran’s affairs, though Chi-Square analysis revealed similar findings regarding
the effectiveness of telephone reminders.
Text reminders. Multiple studies have investigated the effectiveness of text-based
appointment reminders. Boksmati, Butler-Henderson, Anderson, and Sahama (2016) conducted
a meta-analysis of text reminders that included 28 studies. All included studies found that text
EVALUATION OF APPOINTMENT REMINDER SYSTEM
17
message reminders improved appointment adherence. Most reminders were sent within 48-hours
of the scheduled appointment and 75% of the settings sent only one reminder. Most (88%) of the
reminders were generic rather than personalized. These researchers did not find any subgroup
differences for timing, number, or type (automated or personal) of reminders. This may be
related to challenges in comparing the different study designs and the small number of studies
that implemented multiple reminders and/or live reminders. A systematic review by Robotham,
et. al., (2016) had similar findings. The authors’ meta-analysis of 26 articles found that patients
receiving text reminders were 23% more likely to attend appointments than patients who
received no reminder.
Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) conducted a
Cochrane Review of eight RCTs. They found moderate evidence that text reminders improved
appointment attendance compared to no reminders. They also found moderate evidence that
telephone and text reminders had similar outcomes. A systematic review of 60 studies analyzed
the use of text messages to improve various health outcomes, including 11 studies investigating
the impact of text messages on appointment attendance (Kannisto, Kolvunen, and Valimaki,
2014). Attendance rates improved in 77% of the studies.
Studies analyzing text reminders were all systematic reviews or meta-analyses of prior
studies. The effectiveness of text reminders for improving appointment attendance was
consistent across studies. It is important to note there is some overlap of studies included within
the reviews. Inclusion criteria and search methods were clearly defined in each study. Authors
consistently used a 95% CI to determine the significance of differences between groups.
Reminder plus. The literature suggests multiple reminders are optimal for improving
adherence with two reminders increasing attendance up to 19% over one reminder. (Robotham et
EVALUATION OF APPOINTMENT REMINDER SYSTEM
18
al., 2016). Two or more telephone contacts or combination telephone/text reminders were most
often evaluated in reminder plus studies. Callinan et al. (2017) found that a personal telephone
call one month prior to recommended follow-up along with a personal telephone reminder the
day prior improved appointment adherence over those participants receiving usual care of a
mailed reminder followed by an automated telephone call the day prior. Clouse, Williams, and
Harmon (2015) used convenience sampling to determine that subjects (N = 18) had increased
appointment attendance following two personal telephone contacts compared to no reminder for
the same time period the year prior. The researchers utilized a “telephone engagement protocol”
process map to study the effectiveness of a personalized phone call about one week prior to the
scheduled appointment, followed with an appointment reminder the day prior to the appointment.
Eighty-percent of patients who received both contacts attended their scheduled appointment,
resulting in a 26% reduction in no-shows compared to the same time the prior year in which no
reminders were provided. In some instances, subjects in the intervention only received one of
the reminders due to technical challenges or issues with contact information. Findings revealed
that receiving both reminders was statistically beneficial over receiving only one of the intended
reminders.
Shah et al. (2016) studied the effectiveness of a live reminder call seven days prior to the
scheduled appointment, combined with usual care of an automated telephone call the day prior to
the appointment. The live call encouraged patients to make concrete planning about their
appointment such as considering transportation and what they were scheduled to be doing
immediately before their appointment. The no-show rate was significantly lower in the
intervention group, 22% compared to 29% in the control group.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
19
Lin et al. (2016) found that participants receiving text reminders three days prior to
appointments along with the usual care of day prior automated telephone reminder messages had
a 23.5% no-show rate compared to 38.1% no-show rate for participants receiving only the day
prior automated call. Additionally, participants in the intervention group were more likely to
cancel or reschedule their appointments, allowing the opportunity for other patients to be
scheduled into those times.
Percac-Lima et al. (2016) investigated the impact of text reminders sent seven and one
day prior to the scheduled appointment in addition to the usual care of a live telephone reminder
two days prior compared to usual care only. In the intention to treat (ITT), 17% of patients did
not have a mobile number listed in the EMR which perhaps contributed to a lack of significant
findings between the two groups. When the authors completed a matched-control analysis
including only those who agreed to receive the text reminders, there was a 32% reduction in
missed appointments.
Reminder plus was the most frequently studied intervention in this literature review. A
combination of systematic reviews, RCTs, and convenience sampling were reviewed to
determine the effectiveness of these interventions. Sample sizes ranged from N = 17 to N =
8,425. A CI of 95% was the standard for analysis. Chi-square analysis and flow-sheets were
also used for analyzing data. Researchers consistently found that reminder plus significantly
improved appointment attendance compared to single or no reminders. Several of these studies
reported that incorrect patient contact information was a concern and they emphasized the
importance of maintaining up-to-date patient records.
Patient preference. An awareness of patient preference for receiving appointment
reminders helps guide the development of an agency’s reminder system. Telephone and text
EVALUATION OF APPOINTMENT REMINDER SYSTEM
20
message reminders have each been found useful with perhaps a modest preference towards
telephone reminders when researchers have directly compared the two (Robotham et al., 2016).
Crutchfield and Kistler (2017), used convenience sampling to complete a survey of adults
(N = 215) through a national survey company. This online survey of appointment reminders
found that participants preferred one email, phone, or text reminder – in that order of preference -
no more than six days prior to the scheduled appointment. Finkelstein, Liu, Jani, Rosenthal,
Poghosyan (2013) surveyed adults (N = 161) using a cross-sectional study design to assess
preferences for five different appointment reminder systems. Telephone and text reminders both
increased appointment adherence though voice messages left on a mobile or home phone number
were significantly preferred over text messages. Patients reporting more familiarity and usage of
text messaging preferred text-based reminders over voice messages.
Tofighi, Grazioli, Bereket, Grossman, Aphinyanaphongs, and Lee (2017) conducted a
six-month prospective cohort study to obtain patients’ opinion of an appointment reminder
system in an office-based Buprenorphine program. Using a convenience sample (N = 93) the
researchers implemented an appointment reminder system that sent text messages seven, four,
and one day prior to the scheduled appointment. Participants in this study preferred text
messaging to phone reminders. Nearly all participants, 97%, stated the reminders helped them to
attend appointments and 100% of participants believed that all patients should receive text
reminders. A key finding from this study was the frequent turnover of mobile phone contact
information, highlighting the importance of regularly updating patient records.
Studies investigating patient preference used observational design and descriptive
statistics to collect data. The findings were mixed, with one study each reporting telephone, text,
or email as the preferred contact method. Each study sample had the potential for selection bias.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
21
One sample was of opiate dependent adults, a second sample was primarily low-income women,
and the third sample was highly educated and predominantly Caucasian. None of the studies
applied their findings to determine if incorporating patient preference into the reminder system
would actually increase attendance rates.
Limitations
A primary limitation of this literature review is the rapid advancement of technology
along with ever-increasing patient access to technology. Several of the systematic reviews and
meta-analyses included studies dating back as far as 2005. The changes in technology during
that time might render different results if the studies were conducted today. To maintain current
evidence-based findings, additional studies of appointment reminder systems should be
conducted to determine the results in today’s technological world.
The studies contained in the meta-analyses had varying outcome measures and
interventions making it challenging to identify one best method for an appointment reminder
system. Several of the studies acknowledged that inaccurate or incomplete patient contact
information impacted participants’ ability to receive the intended intervention. Many of the
interventions had no method of tracking if the reminder was actually received. It is unknown
how much these missed interventions impacted results. However, it does provide an opportunity
for clinicians and administrators to be reminded of the importance of maintaining accurate
patient information. A number of studies reported on the usefulness of “reminder-plus” methods
though there is no clear indication of what additional reminder methods are most beneficial.
Though some studies investigated patient preferences for receiving reminders, those
studies did not in turn investigate if appointment adherence improved when people received their
preferred reminder. As a result, clinicians must remain cautious of basing their reminder systems
EVALUATION OF APPOINTMENT REMINDER SYSTEM
22
solely on expressed preferences. The majority of the studies investigated appointment adherence
over a limited time-period. One cannot conclude that the observed improvement in adherence
rates will continue over time. It is possible that the increase was related to the novel intervention
with patient response decreasing over time.
Conclusions
Multiple studies support the effectiveness of appointment reminder systems for
improving appointment adherence. The focus of these studies has evolved as more modalities
for reminder systems become available. Early studies investigated the benefit of mailed
reminders whereas in the current research, the focus is primarily on telephone and text-based
reminders with mailings being either obsolete or part of a supplemental reminder system.
While the benefit of text-message reminders is comparable to that of telephone
reminders, there is perhaps a slight patient preference towards live telephone reminders. Though
no exact formula has been identified as the most effective reminder system, a total of two
reminders consisting of either a text and/or telephone contact appears to be the most effective
and preferred modality. Reminders sent between three and seven days and again the day before
the appointment were most often studied. Reminders sent more than seven days prior were not
found to be effective. The studies also found that participant age did not impact their ability or
willingness to receive text-based messages. Participants were not usually concerned about
breaches in confidentiality with text reminders. The vast majority of participants approached for
these studies had access to mobile phones, though maintaining accurate patient contact
information was identified as a challenge.
The reviewed studies represent a diverse sample including urban settings, minority
populations, and a variety of clinic settings including primary care, psychiatry, endocrinology,
EVALUATION OF APPOINTMENT REMINDER SYSTEM
23
and ophthalmology. The studies offer consensus that appointment reminders are an effective
method for improving appointment attendance, with text messaging and telephone reminders as
the preferred modalities. In addition to improving attendance rates, reminders also increase
cancellations rates, allowing for waiting patients to be seen in a timelier manner.
Chapter Three: Conceptual Foundation and Theoretical Model
The Institute for Healthcare Improvement (IHI) describes quality improvement (QI) as an
applied science that incorporates expert knowledge with evidence-based improvement methods.
This is a multidisciplinary approach relying on clinical science and systems theory, among other
fields. The first step in QI is to identify the area for improvement followed by identifying how to
determine if a change results in an improvement and outlining what changes can be made to
result in improvement (IHI, 2018). The Center for Public Health Strategy (2018) developed a QI
roadmap which is detailed in figure 1.
This DNP project used process evaluation to addresses the first three sections of this
roadmap. Understanding the current appointment reminder system is the aim. The measure of
improvement is a reduction in no-show rates for scheduled psychiatric appointments within this
population. The ideas for change will be generated based on the information collected by key
informants. Upon completion, the institution will be provided with detailed recommendations on
how to move forward with the test and spread and sustain components.
Aim:What is the
goal
Measure:Does change
lead to improvement
Ideas:What
changes can result in
improvement
Test:PDSA Cycle
Spread and Sustain
Figure 1. Quality Improvement Roadmap, Adapted from Institute for Healthcare Improvement/Open Access
EVALUATION OF APPOINTMENT REMINDER SYSTEM
24
Program evaluation is integral to QI. Programs are evaluated at varying stages, from
their development to their ultimate impact. Confirming a program is implemented as intended,
known as a process evaluation, is a crucial QI measure. The Center for Disease Control (2018)
describes a process evaluation as a method for measuring whether a program has been
implemented as prescribed. In his book, Theory Driven Evaluations (1990), Chen defines the
program as intended as normative theory. Causative theory is defined as the way the program is
actually delivered. A process evaluation provides the opportunity to identify discrepancies
between the normative and causative theories within a program and identify opportunities for
improvement. Evaluations address the who, what, when, and where of a program. They provide
information on whether a program is effective and accessible to the target population while also
identifying potential problems with a program’s implementation. A well-executed process
evaluation provides stakeholders with a comprehensive, detailed understanding of a program and
provides information to aid in program improvements (CDC, 2018).
Theoretical Model
Process mapping, also referred to as flow-charting, is a component of process evaluation
and QI. Key informants are asked to provide detailed information about the process of interest.
This mapping uses a visual representation to describe the step-by-step procedures. Process
mapping provides an understanding of how process actually works and gives insight on how the
process can be improved. Additionally, process mapping helps to clarify and define complex
procedures and identify unnecessary, duplicate, or problematic steps. This framework allows
team members to achieve a shared understanding of the current process and work together on
their goals for process improvement (IHI, 2018). The final product is displayed in a Cause and
EVALUATION OF APPOINTMENT REMINDER SYSTEM
25
Effect diagram such as an Ishikawa – or fishbone – diagram which provides a concise, visual
representation of findings (Agency for Healthcare Research and Quality, 2018).
Conclusions
Process evaluation is a cornerstone of the QI process, providing a structured approach
for gaining an accurate, comprehensive understanding of an existing program. The evaluator can
determine if a program is being carried out as prescribed and compare the current process to
evidence-based best practices. Incorporating evidence-based practice is a hallmark of nursing
care. The Institute of Medicine’s (IOM) pioneering report, Crossing the Quality Chasm,
identifies evidence-based practice as a core competency of medical practice, with a goal that
90% of clinical decisions be evidenced-based by the year 2020 (IOM, 2001). The benefits of
evidence-based practice include improved patient care, improved health outcomes, and lower
healthcare costs (Melnyk, Fineout-Overholt, Gallagher-Ford, & Kaplan, 2012).
The objective of this process evaluation was to gain a full understanding of the
institution’s appointment reminder system for perinatal psychiatric appointments. Evidence-
based recommendations improving the reminder process were provided to executive staff. The
intended goal is improved access to psychiatric care for perinatal women.
Chapter Four: Project Design
Project Implementation Steps
A well-implemented QI project requires the appropriate selection of the participants,
including all relevant key stakeholders. Staff and clinicians with an active, direct role in the
appointment scheduling and reminder process were identified by the service line director. A
working group of five key informants convened during a two-hour “working lunch” meeting to
discuss the appointment reminder system. A structured meeting agenda identified the
EVALUATION OF APPOINTMENT REMINDER SYSTEM
26
information to be gathered to ensure the meeting objectives were achieved. Flip-charts were
used to capture the data. Process maps of the intended system and the delivered system were
constructed. An Ishikawa diagram was used to visually represent the root cause analysis of
factors contributing to the high no-show rate. Findings were compared to current EBP for
optimizing appointment attendance through reminder systems. Recommendations based on
findings were submitted to the institution in an Executive Summary format. The final project
and outcomes were presented to Drexel University project committee members as per
requirements of the university. The finalized paper detailing the project in its entirety was
submitted to the Drexel University Library archives.
Practice Setting
The process evaluation was conducted in an outpatient obstetrics clinic located within a
large, urban, teaching hospital in Philadelphia. The outpatient obstetrics clinic offers co-located
psychiatric services by a psychiatric nurse practitioner one day per week. The women served
within this clinic are predominately minority women under age 25 with lower socioeconomic
status who are Medicaid eligible. Women are referred to psychiatry directly from the obstetrics
clinic, usually from their obstetrics provider or via a Centering Pregnancy/Parenting program
located within the institution. Centering referrals are typically made by social workers or nurses.
Centering Pregnancy is a program in which pregnant women meet as a group to complete their
prenatal medical appointments and also learn about topics such as nutrition, lactation and self-
care (Centering Healthcare Institute, 2018). To be eligible for the Centering program, the
pregnancy must be considered high-risk due to either comorbid medical conditions - including
depression - or other risk factors such as substance use, poor access to prenatal care, or poor
nutrition.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
27
Key Informants
Staff and clinicians identified by the service line director as performing duties directly
related to the referring, scheduling, and reminding of perinatal psychiatric appointments were
invited to voluntarily participate. The panel was comprised of a multidisciplinary group
including social work, nursing, psychiatry, administration, and call center staff. A total of five
key-informants attended.
Practice Change/Quality Improvement Design
This process evaluation was developed using the CDC’s Framework for Program
Evaluation (1999). This framework identifies the following six-steps for a thorough process
evaluation: engage stakeholders; describe the program; focus the evaluative design; gather
credible evidence; justify conclusions; and share lessons learned. As part of the evaluation,
participants completed process mapping to compare the intended delivery system to the actual
delivery system. The 30 standards of effective evaluation identified by Joint Committee on
Standards for Educational Evaluation guided the integrity of this project (Yarbrough, Shulha,
Hopson, & Caruthers, 2011). These standards require that utility, feasibility, propriety, accuracy,
and accountability be considered in the development and implementation of the evaluation.
Data Collection Tool
The meeting agenda was developed and a copy was provided to all participating
volunteers. A semi-structured interview protocol with probes was used to ensure the objectives
were met (Appendix B). To fully understand the appointment reminder process and identify
possible variations in the process, the appointment referral and scheduling process was
incorporated into the interview protocol for discussion.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
28
Data Collection Methods & Evaluation
A group of five key informants were invited to participate in this two-hour work group
meeting. Food was provided. The meeting followed the agenda using the semi-structured
interview protocol to develop a process map providing a clear understanding of the intended
appointment reminder process compared to the delivered reminder process. Key questions
included: what is the reminder type (text, phone, mail); is the reminder automated or personal;
number of reminders; when is reminder sent; what is frequency/method for updating patient
records; is contact/non-contact tracked?
Flip-charts were used to assemble the data and construct a process map based on
responses. The final agenda item allowed opportunity to confirm that the process map accurately
reflected the key informant’s descriptions and input. Using the process map, current practice
was compared to the known evidence-based practices for optimizing appointment attendance.
An Ishikawa diagram was constructed to visually present the root cause analysis. An Executive
Summary including recommendations for implementing and evaluating the suggested changes to
the current reminder system process was presented to the program’s service line director.
Institutional Review Board Preparation
The necessary IRB approvals were obtained from the participating institution and Drexel
University. This project was considered exempt and was approved based on a Letter of
Determination between the institution and Drexel University. Key informants were purposively
selected via the obstetrics department. All participation was voluntary. No identifying
information or patient records were accessed. The risks involved in this DNP project were
minimal.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
29
Conclusion
This DNP QI project was based on established process evaluation criteria, incorporating
process mapping to provide a rich understanding of the current appointment reminder system.
Purposively selected key informants participated in a work group meeting to develop a thorough
understanding of the reminder system for perinatal psychiatric appointments within this urban
obstetrics clinic.
Chapter Five: Implementation Process and Procedures
This evidence-based QI project was designed to optimize the current appointment
reminder system within a perinatal psychiatric service. The primary implementation was a two-
hour work group meeting with five key informants with direct knowledge of the appointment
scheduling and reminding system. The intended outcome was an increase in appointment
adherence and patient access to care.
The implementation stage of this QI project involved the collaboration of a
multidisciplinary team within the institution. Medical providers from nursing and medicine,
social workers, administrators, technology support, and administrative staff each had integral
roles. Participants from social work and nursing identified the high no-show rate as a primary
concern during an initial information gathering meeting. This information was used to aid in the
development of the resulting evaluation of the appointment reminder system.
Administration identified the appropriate participants, coordinated scheduling, and
secured the meeting location for the work group. An invitational email describing the purpose of
the work group meeting was developed along with a meeting agenda. A semi-structured
interview protocol was constructed based on the CDC’s Framework for Process Evaluation
(1999).
EVALUATION OF APPOINTMENT REMINDER SYSTEM
30
Letters of Permission
A letter of permission authorizing this DNP QI project was provided by the institution’s
Women and Children’s Service Line Director. No other permission letters were required for the
implementation of this project. The DNP project was approved as exempt based on a Letter of
Determination between the two institutions.
Resources
This project required minimal use of resources. The primary required resource was
access to the institution’s internal calendar system to aid in scheduling the work group and
securing meeting space. Administrative staff within the institution agreed to facilitate scheduling
the work group and requesting the necessary meeting space. The practice manager and lead
statistician provided information regarding no-show rates and general demographic data for the
practice.
Conclusion
The development of this DNP project occurred over the course of a 12-month period.
Once program staff identified high no-show rates as a primary concern within the perinatal
psychiatric service, efforts began to construct a relevant DNP project which resulted in this
evidence-based evaluation of the appointment reminder system. The desired outcome was to
increase appointment attendance within the perinatal psychiatric service. The required IRB
protocols were followed to ensure the protection of participants. The participating institution
was provided with an Executive Summary of findings along with detailed suggestions on how to
implement and evaluate the proposed program changes.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
31
Chapter Six: Evaluation and Project Outcomes
Key Informants
The department administrator identified staff and clinicians most closely associated with
appointment scheduling and reminding. Five key informants were selected. Invited participants
included: the call center manager responsible for the agency-wide appointment reminder system;
the practice manager responsible for scheduling initial appointments within the obstetrics clinic
and for monitoring appointment cancelations and no-shows; a registered nurse responsible for
referring and scheduling within the obstetrics clinics; a social worker responsible for referring
and scheduling within the centering pregnancy/parenting program; and the psychiatric nurse
practitioner serving as the primary psychiatric provider for this population of women and also
manages the scheduling of all follow-up psychiatric appointments. All five invited participants
attended the two-hour meeting.
Methods
Upon identification, the five selected key informants were invited to attend the work
group meeting via the agency’s internal meeting scheduling program. A detailed email
describing the purpose of the meeting was sent to each invitee. The meeting was held in a
conference room within the institution. Lunch was provided. Participants received a $10 café
gift card as a thank you for their time. Participants introduced themselves and described their
various roles. The semi structured interview protocol was followed and key data were captured
on flip charts. The purpose of the interview was to: understand the intended appointment
reminder process; understand the actual process; probe regarding challenges identified in the
literature such as the process for updating patient contact information or if the system confirms if
reminders are received; and identify challenges and suggestions for improvement. Upon
EVALUATION OF APPOINTMENT REMINDER SYSTEM
32
conclusion of the meeting, the data were reviewed with the key informants to ensure
understanding and accuracy.
Data Analysis
Data were organized and analyzed based on the structure of the interview: outcomes
related to intended process; outcomes related to actual process; outcomes related to identified
challenges. The data were summarized to offer a comprehensive understanding of the
appointment reminder system as prescribed and as delivered. The reminder system was
compared to evidence-based practices found in the comprehensive literature review. Relevant
findings were highlighted. No-show rates for patients scheduled via the OBGYN clinic and
Centering Pregnancy/Parenting were provided by the lead statistician.
Results
This perinatal psychiatric service has a no-show rate of 58% for patients scheduled
directly from OBGYN and 61% for patients scheduled from Centering Parenting/Pregnancy,
averaging to a 60% no-show rate. The overall OBGYN clinic no-show rate is 35%. The
completion of this process evaluation provided a comprehensive understand of the appointment
reminder process for the psychiatric service with the goal of reducing this particularly high no-
show rate.
Televox is the software use for the appointment reminder system. All patients receive
one automated reminder 48 hours prior to their scheduled appointment. Patients may choose for
reminders to be via telephone or text. Text is the method most often preferred by patients.
Televox reminders allow patients to confirm or cancel their appointments. Televox then
provides a daily report to schedulers and managers which notes if patients confirm or cancel
appointments. If a patient selects “cancel”, a scheduler must review this and manually remove
EVALUATION OF APPOINTMENT REMINDER SYSTEM
33
the appointment from the electronic medical record (EMR) as there is no direct link between
Televox and the institution’s scheduling software.
Televox does not have a process for routinely updating patient contact information. The
system does not confirm receipt of the reminder. The only error message Televox provides is if
a text message is sent to a landline. This will be reflected in the above-mentioned report,
becoming the scheduler’s responsibility to reconcile the contact information.
Perinatal psychiatric appointments are scheduled in one of three ways, see figure 2.
Initial appointments scheduled within the outpatient OBGYN clinic follow the above reminder
system protocol with no noted deviation. The schedulers reported that the initial scheduling
process includes confirming patient contact information and asking them how they prefer to be
reminded about appointments – text or telephone. Beyond this, there is no process for routinely
updating patient information. One scheduler noted that patients are often asked “is everything
the same” rather than having the scheduler provide the information on file to confirm accuracy.
Initial appointments scheduled within the Centering Pregnancy/Parenting group follow
the above reminder system protocol though there are also several informal additions to the
reminder process. The social worker scheduling the appointment sends a personal reminder text
to patients the day prior to the appointment. The social worker also follows up with patients if
they miss their appointment and will aid them in rescheduling. Considering this is not a
standardized component of the reminder system, it is important to note that this additional
contact cannot be guaranteed or confirmed for each appointment. There is no formal system for
updating patient contact information though the frequent contact between patients and providers
may offer more opportunity for updating contact information. The psychiatric nurse practitioner
manages the scheduling of all follow-up appointments. In this scenario, the Televox protocol is
EVALUATION OF APPOINTMENT REMINDER SYSTEM
34
followed with no noted deviations. The psychiatric nurse practitioner does not typically confirm
contact information.
A crucial finding was the realization that the schedule is not being updated to reflect
when patients select the “cancel” option via the Televox reminder. This information is available
in the daily Televox report but there is no assigned staff to review the report and update the
schedule accordingly. This results in appointment openings going unfilled rather than being
offered to patients in need. This also results in patients being erroneously categorized as no-
shows.
Initial Appointment Scheduled via Outpatient
OB Clinic Scheduled By Practice
Manager or RN
Contact Information Confirmed at
Scheduling of Initial Appointment
One Automated Televox Reminder 48
Hours Before Appointment
Initial Appointment Scheduled via Centering
ProgramScheduled by Social
Worker
No Formal Confirmation of
Contact Information Though Frequent
Contact With Patient
One Automated Televox Reminder 48
Hours Before Appointment.
Informal Reminder via Social Worker
Follow-up Appointments Scheduled by PMHNP No Confirmation of Contact information
One Automated Televox Reminder 48
Hours Before Appointment
Figure 2. Scheduling Process for Perinatal Psychiatric Appointments
EVALUATION OF APPOINTMENT REMINDER SYSTEM
35
The psychiatric service within this OBGYN clinic has a no-show rate ranging from 58%-
61% depending on the referral source. The total number of no-shows from January until August,
2018 is 274. While many factors contribute to missed appointments, it has been shown that
appointment reminders improve attendance and also increase cancelation rates allowing for
waiting patients to be seen. Improved attendance also has a financial benefit. The institution has
no current data on the financial impact of these missed appointments. Using the formula of a
$200 loss per no-show, it can be estimated that no-show rates within this perinatal psychiatric
service cost an estimated $80,000 annually.
Conclusions
The appointment reminder system appears to be implemented as intended, with one
automated reminder 48 hours prior to the patient’s appointment. No significant barriers or
challenges related to the use of the system were identified. Several opportunities for
improvement were identified and presented to the institution in an Executive Summary Format
60% Average MissedAppointment Rate for Perinatal Psychiatry Appointments
Incomplete Appointment Information
One Automated Reminder
Not checking cancelled
report
System Error
Unknown If Reminder Received
Inaccurate Patient
Information
Reminder does not provide building/suite information
Occasions of reminder havinginaccurate
appointment information (rare)
No structured process forupdating patient information
Televox is unable to confirm receipt
Research suggeststwo reminders
Preference for personalreminders
No staff assigned to check report
Available appointments go unfilled
Inaccurate inflation ofno-show rate
Figure 3. Ishikawa Fishbone Diagram of Root Cause Analysis
EVALUATION OF APPOINTMENT REMINDER SYSTEM
36
(Appendix C). The reminder protocol differs from evidence-based recommendations which
suggests two reminders are preferable. The institution may benefit from adding a second
reminder within seven days of the appointment. Evidence also indicates a preference towards
personalized reminders rather than automated. Given the exceptionally high no-show rate of this
service, having one reminder be a personalized phone call might prove advantageous. It must be
noted than no difference was found in the no-show rate for patients referred via the Centering
Pregnancy/Parenting program despite the report of additional reminders in this program. It is
difficult to determine the relevance of this considering those additional contacts are not
formalized and those patients are typically higher-risk.
Inaccurate patient contact information is a main barrier for optimizing appointment
reminder systems. Implementing a standard procedure for updating patient contact information
increases the likelihood that reminders are received. The most notable finding is the failure to
update the schedule to reflect cancelations. This is an inefficient use of provider time. Patients in
need are left waiting unnecessarily while the no-show rate of the perinatal psychiatric service is
inaccurately inflated. Assigning staff to review the daily Televox report, fill openings, and
accurately reflect if an appointment has been cancelled is crucial to improving the attendance
rate.
Chapter Seven: Implications to Nursing Practice
The foundation of this DNP project is based on the Essentials of Doctoral Education for
Advanced Practice Nurses. Also known as the DNP essentials, these criteria were developed by
the American Association of Colleges of Nursing (AACN, 2006) to provide a clear definition of
the core competencies required for advanced practice nursing. Each of the eight essentials
EVALUATION OF APPOINTMENT REMINDER SYSTEM
37
played a crucial role in the development of this DNP project. Upon degree completion, the
essentials continue to guide the DNP trained nurse in their daily practice.
Essential I: Scientific Underpinnings for Practice. Determining a singular definition
for nursing is challenging. Most definitions agree that nursing takes on numerous roles. This is
highlighted in the American Nurses Association (ANA, 2017, para. 1) definition which states,
“Nursing is the protection, promotion, and optimization of health and abilities, prevention of
illness and injury, facilitation of healing, alleviation of suffering through the diagnosis and
treatment of human response. To successfully complete these complex tasks, DNP trained nurses
must become experts in accessing, understanding, evaluating, and implementing the relevant
science and theories in their practice area of expertise.
Applicability to Project. This DNP project was developed using rigorous scientific
principles. After identifying the problem of interest, a thorough review of the literature was
completed to identify best practices for increasing appointment attendance. Research was
critically assessed for quality and relevance using proven critical assessment tools. Trained
librarians assisted in identifying appropriate search terms and keywords to ensure an expansive
literature review. Articles were synthesized to determine best practices.
Implications for practice. Quality nursing practice relies on evidence-based
interventions to reduce practice gaps. The DNP educated nurse is able to identify opportunities
for practice improvement and engage in sound scientific methods to implement and evaluate
quality improvement initiatives. The ability to navigate and critically assess existing literature is
an expectation.
Essential II: Organizational and Systems Leadership for Quality Improvement and
Systems Thinking. Advanced practice nurses often perform their roles in large organizational
EVALUATION OF APPOINTMENT REMINDER SYSTEM
38
settings. The structure of the United States healthcare delivery system has resulted in the call for
nursing education that focuses on meeting the needs of the current system, while also ensuring
safe, ethical, and quality nursing practice (Zaccagnini & White, 2017). The DNP educated nurse
is expected to practice to their full nursing capability while maintaining an awareness of the
work-flow and fiscal aspects of the organization and considering how these areas can be
improved.
Applicability to Project. This QI initiative was initiated within a large organization. A
primary task for successful project implementation was the identification of relevant
stakeholders – key informants – to provide information regarding the appointment reminder
system. Meeting with key stakeholders provided the opportunity to identify systemic challenges
faced by the organization in promoting patient adherence to psychiatric appointments. The
assessment of identified system provided the opportunity to disseminate key findings.
Recommendations for implementing and evaluating a rapid cycle improvement project were
provided to the institution with the purpose of improving appointment attendance and patient
access to care.
Implications for practice. The DNP educated nurse is capable of navigating complex
organizations and identifying the relevant stakeholders for desired QI initiatives. They should
feel competent in evaluating a practice area and developing a rapid cycle improvement project
based on an identified practice gap. The DNP trained nurse is able to present findings to the
stakeholders in a clear and evidence-based manner.
Essential III: Clinical Scholarship and Analytical Methods for Evidence-Based Practice.
The DNP educated nurse develops and carries out sound research methodologies to address
identified practice gaps. The DNP nurse uses this knowledge to successfully translate evidence-
EVALUATION OF APPOINTMENT REMINDER SYSTEM
39
based findings into practice, evaluating the results along the way. Translating research into
practice is a cornerstone of this DNP essential.
Applicability to Project. Upon identifying the high no-show rate as a practice area of
concern, a PICOT question was constructed to aid in guiding the literature review. The PICOT
format allows nurses to systematically formulate a clinical question and to gather the best
evidence to address the question (Echeverria & Walker, 2014). The literature was synthesized to
focus on the appointment reminder system, directing this project towards a specific area of
evaluation. The evaluation was designed using a known scientific framework for QI
implementation (CDC, 1999). A primary outcome of the evaluation was the dissemination of
results to key stakeholders. This process promoted clinical scholarship and identified evidence-
based practice level changes to reduce no-show rates and increase patient access to care.
Implications for practice. The scientific application of a PICOT questions allows the
DNP trained nurse to critically appraise the literature related to an identified practice gap. The
nurse successfully constructs, implements, and evaluates an evidence-based quality improvement
intervention. Disseminating findings to relevant stakeholders facilitates the translation of
evidence into practice.
Essential IV: Information Systems/Technology and Patient Care Technology for the
Improvement and Transformation of Health Care. The AACN states that, “DNP graduates
are distinguished by their abilities to use information systems/technology to support and improve
patient care and healthcare systems…” (AACN, 2016, p. 12), emphasizing the importance of
technology in providing safe, quality patient care. This essential is especially relevant to this
DNP project considering the automated appointment reminder system, Televox, was the primary
focus of the evaluation.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
40
Applicability to Project. A comprehensive literature review of relevant electronic
databases as well as use of the World Wide Web for ongoing online inquiry to identify additional
scientific based resources and gray literature set the foundation for this DNP project. An
understanding of the Televox technology was necessary to construct an interview protocol with
relevant questions with adequate probes. The outcomes of this evaluation include
recommendations for optimizing this technology to improve patient access to care.
Implications for practice. The DNP trained nurse must be competent in the utilization
of technology to promote patient care. This includes electronic medical records (EMR), patient
portals, and other modes of technology intended to advance patient care. These systems often
house data that offers insight into the current state of practice. Technological systems and EMRs
are primary tools used for identifying potential practice areas in need of improvement. The DNP
nurse must be competent in extracting and analyzing this data to promote QI initiatives.
Essential V: Healthcare Policy for Advocacy in Health Care. The American Nurses
Association (ANA) considers patient advocacy as a “pillar” of nursing (ANA, 2018, para. 1).
Healthcare policy has the power to promote or impede the healthcare services provided to
patients. The DNP trained nurse can expect to be at the forefront of patient advocacy (AACN,
2006).
Applicability to Project. Patient advocacy was a consideration throughout this DNP
project, beginning with careful attention to the IRB protocol of both participating institutions.
This project was guided by the intent to improve access to psychiatric care for
underserved/perinatal women. A primary outcome of this project involves advocating for
institutional changes in the current reminder system to increase patient access to care.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
41
Implications for practice. Maintaining an awareness of deficits within the healthcare
system is an important component of the DNP role. Through proactive participation in policy
development the DNP trained nurse contributes to the reduction of health disparities and
improved healthcare access. Opportunities for advocacy range from the institutional to the
international level (AACN, 2006).
Essential VI: Interprofessional Collaboration for Improving Patient and Population
Health Outcomes. The DNP trained nurse assumes a leadership role that facilitates
interprofessional collaboration. Healthcare organizations are challenged with the need to
improve the quality of care by instituting standards of care and developing health policy while
also maintaining fiscal responsibility. The DNP trained nurse is qualified to aid organizations in
achieving these goals (AACN, 2006). Considering that multiple disciplines are involved with
patient care, developing effective interprofessional collaboration is necessary to promote
efficient, quality care (Poston, Haney, Kott & Rutledge, 2017).
Applicability to Project. This DNP project brought together participants from pediatrics,
obstetrics, psychiatry, social work, nursing, and information systems. Program administrators
participated in the project development and implementation. This DNP project provided the
opportunity for communication about the appointment reminder process across disciplines.
Considerations specific to psychiatry were highlighted and addressed. The collaboration of
psychiatry with obstetrics is an important aspect of this DNP project. An initiative to improve
the show-rate for psychiatry results in more efficient use of the psychiatric provider’s time and
increases the likelihood that this important collaboration between behavioral health and women’s
health continues.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
42
Implications for practice. Research indicates that embedded, collaborative care promotes
increased access to care and improved health outcomes (Van Eck, et al., 2018). Promoting
collaborations through QI initiatives facilitates an interdisciplinary approach to patient care. An
understanding of the varied roles and work flow dynamics across disciplines allows the DNP
trained nurse to identify practice gaps and implement work-flow improvements. Effective
communication and leadership skills are necessary to engage the interprofessional teams
(AACN, 2006).
Essential VII: Clinical Prevention and Population Health for Improving the
Nation’s Health. Health promotion and disease prevention are at the heart of nursing. Nursing
practice is directed by an awareness of population health and social determinants of health.
Engaging with patients with cultural and emotional sensitivity fosters relationships and
positively impacts care (AACN, 2006). Healthy People 2020 identifies health equity, reduced
disparities, health promoting environments, and the promotion of health behaviors as their
fundamental goals (Office of Disease Prevention and Health Promotion, 2018).
Applicability to Project. Untreated mental health has a significant impact on the person
and their community. Social determinants of health such as housing, community support, and
employment opportunities are known to increase the risk for depression. The overarching goal
of this DNP project is to increase access to mental health care for an underserved population.
The evaluation of this specific care delivery system – the appointment reminder system –
highlighted issues such as patient literacy, access to telephones, and knowledge of technology.
Implications for practice. Quality nursing care engages in both treatment and
prevention. It is necessary to treat the entire patient and consider their environmental and social
EVALUATION OF APPOINTMENT REMINDER SYSTEM
43
risk factors. This awareness allows the nurse to develop a treatment plan that includes patient
education and prevention strategies for health promotion.
Essential VIII: Advanced Nursing Practice. Advanced nursing practice encompasses each
of the DNP essentials. The advanced practice nurse is expected to have highly developed
assessment skills and sound knowledge of pathophysiology and pharmacology. The APN is
expected to provide quality, evidenced-based, patient-centered care.
Applicability to Project. Advanced nursing practice was present throughout this DNP
project. The PICOT question and literature review were developed using principles of nursing
science. Relationships were established to support the development and implementation of this
QI initiative. The successful completion of this DNP project relied on the engagement of
interdisciplinary participants. Th received evidence-based recommendations to aid in improving
mental health access for postpartum women. This project provides an example of nursing
identifying a practice gap and utilizing advance practice nursing skills to translate evidence into
practice.
Implications for practice. DNP educated nurses are experts in the multiple facets of nursing
practice. They possess the skills to formulate science-based practice questions. The DNP nurse
designs, implements, and evaluates interventions to address these practice questions. Effective
leadership and evidence-based, patient-centered care incorporates the tenets of advanced practice
nursing. This allows for ongoing positive contributions to the ever-growing body of nursing
science. Modeling high-quality nursing practice has a positive influence on nursing and the
multiple other disciplines working closely with nursing.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
44
Chapter Eight: Project Implications and Conclusions
This DNP project highlights potential deficits of a large organization’s appointment
reminder system. Several suggestions for improving the current reminder system were provided
to the institution based on the outcomes of this evaluation. Implementing the recommendations
may increase patient access to care and improve the fiscal outcomes of the obstetrics department.
Strengths
An interdisciplinary approach was used to identify the practice concern of no-show rates
within the perinatal psychiatry service. This project is important for nursing practice considering
it involves patient access to care and advocating for patients who are often underserved.
Principles of nursing science were applied to develop the PICOT question and guide the
literature review. The process evaluation was developed using evidence-based techniques for QI
initiatives. The specific focus on the appointment reminder system allowed for a highly specific
evaluation resulting in targeted suggestions for improvement. The finding that cancelled
appointments are not currently noted in reminder provides a notable opportunity for change
Limitations
The appointment reminder system is only one possible contributor to high appointment
no-show rates. Though implementation and evaluation of the recommendations was beyond the
scope of this DNP project, it will be important for future work to evaluate if the suggested
changes actually improve appointment attendance. The institution is encouraged to implement
the rapid cycle improvement strategy as suggested in the Executive Summary. There is no
specific information available on the financial impact of these missed appointments. This
evaluation provides an estimate based on using a monetary amount cited in the literature. With a
no-show rate as high as 60%, awareness of the fiscal impact is important.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
45
Significance and Implications
Untreated perinatal mental health is a known practice gap that impacts the mother, the
child, and the community. The current no-show rate for the perinatal psychiatric service puts the
collaboration between psychiatry and obstetrics at risk. Considering this is the most requested
referral within the obstetrics clinic, the need and desire for this service is evident. Key findings
from this process evaluation have the potential to increase appointment attendance and ensure
this important collaboration continues. The findings from this process evaluation may
potentially benefit the institution’s entire Televox system considering the current appointment
reminder system is standardized across all hospital departments.
Conclusions
This DNP project addressed an important practice gap within a large hospital setting.
The project was conducted using comprehensive research methods and nursing science. The
DNP essentials guided the project’s development and implementation. This project provides an
example of translating research into practice with the potential to increase patient access to care,
optimize provider resources, and improve financial outcomes.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
46
References
Andrews, P., Thomson, A. (2009). Depression’s evolutionary roots. Scientific American.
Retrieved from: https://www.scientificamerican.com/article/depressions-evolutionary/
Agency for Healthcare Research and Quality (2018). Plan Do Study Act Cycle. Retrieved from:
https://innovations.ahrq.gov/qualitytools/plan-do-study-act-pdsa-cycle
American Association of Colleges of Nursing (2006). The Essentials of Doctoral of Education
for Advanced Nursing Practice. Retrieved from:
http://www.aacnnursing.org/Portals/42/Publications/DNPEssentials.pdf
American Nurses Association (2018). Advocacy. Retrieved from:
https://www.nursingworld.org/practice-policy/advocacy/
American Nurses Association (2018). Scope of Practice. Retrieve from:
https://www.nursingworld.org/practice-policy/scope-of-practice/
American Psychological Association (2017). What is postpartum depression and anxiety?
Retrieved from: http://www.apa.org/pi/women/resources/reports/postpartum-
depression.aspx
American Psychiatric Association, Center for Workplace Mental-Health (2017). Quantifying the
cost of depression. Retrieved from: http://workplacementalhealth.org/Mental-Health-
Topics/Depression/Quantifying-the-Cost-of-Depression
Boksmati, N., Butler-Henderson, K., Anderson, K., Sahama, T. (2016). The effectiveness of
SMS reminders on appointment attendance: A meta-analysis. Journal of Medical Systems,
doi: 10.1007/s10916-016-0452-2
Byatt, N., Simas, T., Lundquist, R., Johnson, J., Ziedonis, D. (2012). Strategies for
improving perinatal depression treatment in North American outpatient obstetric
EVALUATION OF APPOINTMENT REMINDER SYSTEM
47
settings. Journal of Psychosomatic Obstetrics and Gynecology, 33(4), 143-161.
Byatt, N., Biebel, K., Friedman, L., Debordes-Jackson, G., Ziedonis, D., & Pbert, L. (2013).
Patient’s views on depression care in obstetric settings: How do they compare to the views
of perinatal health care professionals? General Hospital Psychiatry, 35, 598-604.
Byatt, N., Levin, L., Ziedonis, D., Moore-Simas, Allison, J. (2015). Enhancing participation in
depression care in outpatient perinatal care settings. Obstetrics and Gynecology, 126(5),
1048-1058.
Callinan, C., Kenney, B., Hark, L., Murchison, A., Dai, Y., Leiby, B., Mayro, E., Bilson, J.,
Haller, J. (2015). Improving follow-up adherence in a primary eye care setting. American
Journal of Medical Quality, 32(1), 73-79.
Center for Disease Control (1999). Framework for Program Evaluation in Public Health.
Retrieved from: https://www.cdc.gov/mmwr/PDF/rr/rr4811.pdf
Center for Disease Control (2017). Depression among women. Retrieved from:
https://www.cdc.gov/reproductivehealth/depression/index.htm
Center for Public Health Quality (2018). Step by Step Guide to Implement Quality Improvement.
Retrieved from: http://www.amchp.org/Transformation-
Station/Documents/QI%20Step%20by%20Step%20Guide.pdf
Centering Healthcare Institute (2018). Retrieved from: https://www.centeringhealthcare.org
Chen, H. (1990). Theory-Driven Evaluations. Newbury Park, CA: Sage Publications.
Clouse, K., Williams, K., Harmon, J. (2017). Improving the no-show rate of new patients in
outpatient psychiatric practice: An advance practice nurse-initiated telephone engagement
protocol quality improvement project. Perspectives in Psychiatric Care, 53, 127–134.
Crutchfield, T. M., & Kistler, C. E. (2017). Getting patients in the door: medical appointment
Running head: EVALUATION OF APPOINTMENT REMINDER SYSTEM
48
reminder preferences. Patient preference and adherence, 11, 141.
Echevarria, I. M., & Walker, S. (2014). To make your case, start with a PICOT question.
Nursing2018, 44(2), 18-19.
Field, T. (2010). Postpartum depression effects on early interactions, parenting and
safety practices: A review. Infant Behavior and Development, 33(1), 1-6.
Finkelstein, S. R., Liu, N., Jani, B., Rosenthal, D., & Poghosyan, L. (2013). Appointment
reminder systems and patient preferences: Patient technology usage and familiarity with
other service providers as predictive variables. Health informatics journal, 19(2), 79-90.
Gajwani, P. (2014). Can what we learned about reducing no-shows in our clinic work for you?,
Current Psychiatry, 13(9), 13.
Geier, M., Hills, N., Gonzales, M., Tum, K., Finley, P. (2014). Detection and treatment
rates for perinatal depression in a state Medicaid population. CNS Spectrums, 20, 11-
19.
González, H. M., Vega, W. A., Williams, D. R., Tarraf, W., West, B. T., & Neighbors, H. W.
(2010). Depression Care in the United States: Too Little for Too Few. Archives of General
Psychiatry, 67(1), 37–46. http://doi.org/10.1001/archgenpsychiatry.2009.168
Grote, N., Katon, W., Russo, J., Lohr, M.J., Curran, M., Galvin, E., Carson, K. (2015).
Collaborative care for perinatal depression in socioeconomically disadvantaged women: A
randomized controlled trial. Depression and Anxiety, 32, 821-834.
Gurol-Urganci, I., de Jongh, T., Vodopivec-Jamsek, V., Atun, R., & Car, J. (2013). Mobile
phone messaging reminders for attendance at healthcare appointments. Cochrane Database
Syst Rev, 12(12), CD007458.
Running head: EVALUATION OF APPOINTMENT REMINDER SYSTEM
49
Hansotte, E., Payne, S., Babich, S. (2017). Positive postpartum depression screening practices
and subsequent mental health treatment for low-income women in Western countries: A
systematic literature review. Public Health Reviews, 38(3), doi: 10.1186/s40985-017-0050-y
Institute for Healthcare Improvement (2018). Science of Improvement: How to Improve.
Retrieved from:
http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementHowtoImprove.aspx
Institute of Medicine (2001). Crossing the quality chasm: A new health system for the 21st
century. Retrieved from:
http://www.nationalacademies.org/hmd/~/media/Files/Report%20Files/2001/Crossing-the-
Quality-Chasm/Quality%20Chasm%202001%20%20report%20brief.pdf
Kannisto, K., Koivunen, M., Valimaki, M. (2014). Use of mobile phone text message reminders
in health care services: A narrative literature review. Journal of Medical Internet Research,
16(10), doi: 10.2196/jmir.3442
Kaplan-Lewis, E., Percac-Lima, S. (2013). No-show to primary care appointments: Why patients
do not come. Journal of Primary Care and Community Health, 4(4), 251-255.
Kheirkhah, P., Feng, Q., Travis, L. M., Tavakoli-Tabasi, S., & Sharafkhaneh, A. (2015).
Prevalence, predictors and economic consequences of no-shows. BMC health services
research, 16(1), 13. Retrieved from:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4714455/pdf/12913_2015_Article_1243.pdf
Ko, J., Farr, S., Dietz, P., Robbins, C. (2012). Depression and treatment among U.S. pregnant
and nonpregnant women of reproductive age, 2005-2009. Journal of Women’s Health, 21(8),
doi: 10.1089/jwh.2011.3466
Running head: EVALUATION OF APPOINTMENT REMINDER SYSTEM
50
Kopelman, R., Moel, J., Mertens, C., Stuart, S., Arndt, S., O’Hara, M. (2008). Barriers to
care for antenatal depression. Psychiatric Services, 59(4), 429-432.
Lin, C., Mistry, N., Bonch, J., Li, H., Lazebnik, R. (2016). Text message reminders increase
appointment adherence in a pediatric clinic: A randomized controlled trial. International
Journal of Pediatrics, doi: 10.1155/2016/8487378
McLean, S., Gee, M., Booth, A., Salway, S., Nancarrow, S., Cobb, M., & Bhanbhro, S. (2014).
Targeting the Use of Reminders and Notifications for Uptake by Populations (TURNUP): a
systematic review and evidence synthesis. Health Services and Delivery Research, 2(34), 1-
184.
Megnin-Viggars, O., Symington, I., Howard, L., Pilling, S. (2015). Experience of care for mental
health problems in the antenatal or postnatal period for women in the UK: A systematic
review and meta-synthesis of qualitative research. Archive of Women’s Mental Health, 18,
745-759.
Melnyk, B., Fineout-Overholt, E., Gallagher-Ford, L., Kaplan, L. (2012). The state of evidence-
based practice in US nurses. The Journal of Nursing Administration, 42(9), 410-417.
Melnyk, B. (2012). Achieving a high reliability organization through implementation of the
ARCC model for systemwide sustainability of evidence-based practice. Nursing
Administration Quarterly, 36(2), 127-135.
Mitchell, A., Selmes, T. (2007). Why don’t patients attend their appointments? Maintaining
engagement with psychiatric services. Advances in Psychiatric Treatment, 13(6), 423-434.
Moehler, E., Brunner, R., Wiebel, A., Reck, C., Resch, F. (2006). Maternal depressive
EVALUATION OF APPOINTMENT REMINDER SYSTEM
51
symptoms in the postnatal period are associated with long-term impairment of
mother–child bonding. Archives of Women’s Mental Health, 9(5), 273-278.
National Alliance on Mental Illness (2018). Mental Health Facts in America. Retrieved from:
https://www.nami.org/NAMI/media/NAMI-Media/Infographics/GeneralMHFacts.pdf
National Institute of Mental Health, (2018). Statistics. Retrieved from:
https://www.nimh.nih.gov/health/statistics/index.shtml#Intro
Office of Disease Prevention and Health Promotion (2018). Social Determinants of Health.
Retrieved from: https://www.healthypeople.gov/2020/topics-objectives/topic/social-
determinants-of-health
Parikh, A., Gupta, K., Wilson, A. C., Fields, K., Cosgrove, N. M., & Kostis, J. B. (2010). The
effectiveness of outpatient appointment reminder systems in reducing no-show rates. The
American journal of medicine, 123(6), 542-548.
Percac-Lima, S., Singer, D., Cronin, P., Chang, Y., Zai, A. (2016). Can text messages improve
attendance to primary care appointments in underserved populations? Journal of Health
Care for the Poor and Underserved, 27(4), 1709-1725.
The Pell Institute (2018). Using a Logic Model. Retrieved from:
http://toolkit.pellinstitute.org/evaluation-guide/plan-budget/using-a-logic-model/
Poston, R., Haney, T., Kott, K., & Rutledge, C. (2017). Interprofessional team performance,
optimized. Nursing management, 48(7), 36-43.
Robotham, D., Satkunanathan, S., Reynolds, J., Stahl, D., Wykes, T. (2016). Using digital
notifications to improve attendance in clinic: Systematic review and meta-analysis. British
Medical Journal Open, doi: 10.11136/bmjopen-2016-012116
Running head: EVALUATION OF APPOINTMENT REMINDER SYSTEM
52
Scientific American, Editors, (2012). The Neglect of Mental Illness Exacts a Huge Toll,
Human and Economic. Retrieved from: https://www.scientificamerican.com/article/a-
neglect-of-mental-illness/
Shah, et al., (2016). Targeted reminder phone calls to patients at high-risk of no-show for
primary care appointment: A randomized trial. Journal of General Internal Medicine,
31(12), 1460-1466.
Teo, A., Forsberg, C., Marsh, H., Saha, S., Dobscha, S. (2017). No-show rates when phone
appointment reminders are not directly delivered. Psychiatric Services, doi:
10.1176/appi.ps.201700128.
Tofighi, B., Grazioli, F., Bereket, S., Grossman, E., Aphinyanaphongs, Y., Lee, J. (2017). Text
message reminders for improving patient appointment adherence in an office-based
Buprenorphine program: A feasibility study. The American Journal on Addictions, 26, 581-
586.
van Eck van der Sluijs, J. F., Castelijns, H., Eijsbroek, V., Rijnders, C. A., van Marwijk, H. W.
J., & van der Feltz-Cornelis, C. M. (2018). Illness burden and physical outcomes associated
with collaborative care in patients with comorbid depressive disorder in chronic medical
conditions: A systematic review and meta-analysis. General Hospital Psychiatry:
Psychiatry, Medicine and Primary Care, 50, 1-14.
Weinreb, L., Byatt, N., Moore-Simas, T., Tenner, K., Savageau, J. (2014). What happens
to mental health treatment during pregnancy? Psychiatry Quarterly, 85(3), 349-355.
World Health Organization (2017). Depression fact sheet. Retrieved from:
http://www.who.int/mental_health/prevention/genderwomen/en
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). Program evaluation
standards. Thousand Oaks, CA: Sage
EVALUATION OF APPOINTMENT REMINDER SYSTEM
53
Zimmerman, S. (2015). Insight: Tackling healthcare’s costly problem of missed appointments.
Healthcare Finance. Retrieved from: http://www.healthcarefinancenews.com/blog/insight-
tackling-healthcares-costly-problem-missed-appointments
EVALUATION OF APPOINTMENT REMINDER SYSTEM
54
Appendix A
Table of Evidence
Citation Research Design/Level of Evidence
Sampling & Methods
Data Analysis Findings Comments/Limitations
Boksmati, N., Butler-Henderson, K., Anderson, K., Sahama, T. (2016). The effectiveness of SMS reminders on appointment attendance: A meta-analysis. Journal of Medical Systems, doi: 10.1007/s10916-016-0452-2
Level Ib: Experimental Design. Meta-Analysis of RCT and other study designs
Literature review of OvidSP, PubMed, EMBASE and CINAHL with keywords texting, text reminder, text messaging, reminder systems. All articles evaluated SMS system against a control group of no reminder, English language, published between 2005-2015. 303 records found, 28 included in analysis. 13 (46%) RCT, 15
Frequency analysis for country, attendance rate, specialty, SMS timing, study design. Attendance rate was primary outcome. 95% confidence interval, intervention group compared to control group.
SMS reminders improved appointment attendance compared to control group with OR of 1.62. 75% of SMS sent within 48 hours of appointment 75% of settings sent only one reminder, No differences found for age, timing of reminder, or if reminder was personal or generic.
4 studies conducted in United States, majority conducted in the UK.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
55
(53%) observational.
Callinan, C. Kenney, B., Hark, L., Murchison, A., Dai, Y., Leiby, B., Mayro, E., Bilson, J., Haller, J. (2015). Improving follow-up adherence in a primary eye care setting. American Journal of Medical Quality, 32(1), 73-79.
Level Ic: Experimental Design. Randomized controlled trial
Prospective, single-blinded RCT. Patients in Philadelphia eye clinic, randomized to usual care (UC), automated intervention, personal intervention. UC (N:364) received auto call reminder day before appointment, automated intervention (N: 365) received mailed appointment reminder, day prior reminder call and follow up postcard if they missed appointment, personal intervention (N:
Primary outcome measure was adherence to follow-up appointments. 3-month time period, 80% power to detect increase in attendance in either intervention compared to UC. 95% CI, P value of 0.5, Poisson regression model to detect treatment group differences.
Personal intervention more likely to attend appointments (38%) than UC (28%) and automated intervention (30%). No significant difference between UC and automated.
Urban setting, sample primarily African American, primarily Medicaid/Medicare coverage. Maintaining up-to-date patient information was listed as challenge to providing reminders.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
56
366) all of the above plus personal telephone call prior to appointment and if appointment was missed.
Clouse, K., Williams, K., Harmon, J. (2015). Improving the no-show rate of new patients in outpatient psychiatric practice: An advanced practice nurse-initiated telephone engagement protocol quality improvement project. Perspectives in Psychiatric Care, 53, 127-134.
Level IIIe: Observational, analytic design. Convenience sample
Convenience sample of 27 subjects who met criteria, 18 of which consented to participate. 78% female, no formal assessment tools, TEP flowchart used to document each contact, intervention is two patient contacts – brief call which described first visit, personal reminder call day before appointment.
Flow-sheets were used to document each patient contact.
80% of patients who received both contacts attended appointment, attendance rates better for those who received both contacts, overall 7% improvement in show rates for intervention compared to UC. No effect for age or type of insurance.
Small sample, no control group, maintaining up-to-date patient information was listed as a challenge to providing reminders.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
57
Crutchfield, T., Kistler, C. (2017). Getting patients in the door: Medical appointment reminder preferences. Patient Preference and Adherence, 11, 141-150.
Level IVb: Observational/Descriptive Convenience sample - survey
Participants recruited via listserv of university (N=251). Participants completed online survey inquiring about appointment reminder preferences.
Used statistical software for descriptive analyses of which reminder methods were preferred.
Email, phone and text were preferred reminder methods, within one to six days prior to appointment, single reminder was preferred.
Selection bias – mostly Caucasian, highly educated sample.
Finkelstein, S., Liu, N., Jani, B., Rosenthal, D., Poghosyan, L. (2013). Appointment reminder systems and patient preferences: Patient technology usage and familiarity with other service providers as predictive variables. Health Informatics Journal, 19(2), 79-90.
Level IVd: Observational/Descriptive Cross-sectional survey
Assessed patient preference for five different reminder systems. N= 161, urban setting, diverse sample, low-income, convenience sample.
Self-report survey instrument, rank ordered reminder preference from most to least preferred, assessed patient responsiveness to various appointment reminder systems. Descriptive statistics and frequencies were analyzed using SPSS, Friedman nonparametric to assess ranked preferences and Wilcoxon Signed Ranks to assess for significant differences between
Preferences were ranked as: cell phone, home phone, text message, email, direct mail. Familiarity with and usage of technology were indicators of responsiveness to reminders.
78% of sample was female, urban setting, majority Medicaid/Medicare. Self-report convenience sample.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
58
ranks. Effectiveness of and responsiveness to reminder systems were dependent variables.
Gurol-Urganci, I., de Jongh, T., Vodopivec-Jamsek, V., Atun, R., Car, J. (2013). Mobile phone messaging reminders for attendance at healthcare appointments (Review). Cochrane Database of Systematic Reviews, doi: 10.1002/14651858.cd007458.pub3
Level Ia: Experimental Design. Systematic review of RCTs
Cochrane review of MEDLINE, EMBASE, PsycINFO, CINAHL and grey literature. RCTs assessing text messaging as reminder system. Included 8 RCTs with total of 6,615 participants.
GRADEpro software used to assess quality of evidence, RR used for effect measures, meta-analysis to calculate overall effect size.
Moderate quality evidence that text reminders improve appointment adherence compared to no reminders (RR: 1.14) 95% CI. Three studies found text and phone reminders equal. Overall 78% attendance for text, 80% attendance for phone call, 68% attendance for no reminder.
No conclusions on timing of messages or combining reminder methods.
Kannisto, K., Koivunen, M., Valimaki, M. (2014). Use of mobile phone text message reminders in health care services: A narrative literature review. Journal of Medical Internet Research, 16(10), doi: 10.2196/jmir.3442
Level Ib: Experimental Design. Systematic review of RCT and other study designs
Systematic review of PubMed, CINAHL, PsycINFO and other databases, original yield of 911 articles, 60
Identified 14 variables of interest, entered into data extraction grid, quality appraised using Mixed Method Appraisal Tool.
Outcomes improved in 77% of studies, with 11 of those studies investigating appointment attendance.
Diverse methodology and varying quality in studies, frequently changing patient contact information identified as challenge.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
59
included in review. 73% SMS only, 27% SMS/Plus. 35% RCT, 95% quantitative design. Sample size from 4 - 9,959. Looked at various health reminders, not just appointment
Lin, C., Mistry, N., Bonch, J., Li, H., Lazebnik, R. (2016). Text message reminders increase appointment adherence in a pediatric clinic: A randomized controlled trial. International Journal of Pediatrics, doi: 10.1155/2016/8487378
Level Ic: Experimental Design Randomized Controlled Trial/Blinded Assessment
RCT in urban pediatric clinic. Randomized to receive text or not receive text in addition to UC of phone reminder. N = 170. Text sent three days prior to appointment, charts reviewed to determine who attended appointments.
Sampled for 90% power to detect 15% difference between groups, 169 participants were randomized into control or intervention group, 95% CI with p value >0.05 as significant.
Significant difference in no-show rate for control group (38.1%) compared to 23.5% for intervention group, intervention group more likely to keep, cancel or reschedule appointments.
Urban setting, no-show rate remained 24% even with text message.
McLean, S., Gee, M., Booth, A., Salway, S., Nancarrow, S., Cobb, M., Bhanbhro, S. (2014).
Level Ib: Experimental Design.
3 interrelated systematic reviews of
Identified six variables of interest on patient
Timing of reminders from 1-7 days has
Weak evidence that reminder plus (health information in reminder
EVALUATION OF APPOINTMENT REMINDER SYSTEM
60
Targeting the use of reminders and notifications for uptake populations (TURNUP): A systematic review and evidence synthesis. Health Services and Delivery Research, 2(34), doi: 10.3310/hsdr02340
Systematic review of RCT and other study designs
reminder systems, reviewed Cochrane Library, EMBASE and others. 31 RCTs and 11 other systematic reviews were included.
attendance, data extraction tool used to determine patterns and quality of evidence.
same outcome, reminders increase cancellations, to optimize reminders patient information must be up-to-date, information beyond date, time and location is helpful for first appointments.
or dual reminders) improves adherence.
Molfenter, T. (2013). Reducing appointment no-shows: Going from theory to practice. Journal of Substance Use and Misuse, 48(9), 743-749
Level IIc: Quasi-Experimental Design Prospectively controlled study
SAMHSA funded program investigating 67 treatment agencies across 10 states with focus on improving appointment adherence.
Programs implemented practices of their choosing thought to reduce no-show rates, ANOVA analysis separated sites by urban and rural and differences between initial and post no-show rates with descriptive statistics and frequencies for initial appointment and follow-up no-shows.
Significant difference in mean no-show rates post intervention: 16.48%, Reminder calls did improve appointment attendance though it was not a significant finding in this study, creating a welcoming environment, reduced wait times, more capacity and behavioral
Appointment reminders alone may not be adequate.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
61
engagement were most significant interventions.
Parikh, A., Gupta, K., Wilson, A. C., Fields, K., Cosgrove, N. M., & Kostis, J. B. (2010). The effectiveness of outpatient appointment reminder systems in reducing no-show rates. The American journal of medicine, 123(6), 542-548.
Level 2c: Prospective, randomized, parallel design clinical trial.
Random assignment to personal telephone reminder, automated telephone reminder, no reminder. Reminders made three days prior to appointment with continued attempts for following two days if no contact made. N= approx. 3,000 per group based on 80% power.
No-show rate was primary outcome of interest. Analyzed using Intention to Treat analysis, chi-square analysis to determine between group differences using OR and 95% CI.
Significantly higher no-show rate for new patients compared to established patients across all groups (17.7%, 15.9%), No-show rates significantly lower for personal (13.6%), auto (17.3) reminders compared to no-reminder (23.1%).
Patients found reminders helpful but did not remember if reminder was personal or automated.
Percac-Lima, S., Singer, D., Cronin, P., Chang, Y., Zai, A. (2016). Can text messages improve attendance to primary care appointments in underserved populations? Journal of Health Care for the Poor and Underserved, 27(4), 1709-1725.
Level Id: Experimental Design Pseudo RCT
Appointment reminder intervention outside of Boston, MA. Randomized into Intervention = 8,425, Control = 2,679. Intervention
Chi-square analysis was used for between group differences, no show rates compared using logistic regression, 88% power.
With Intention to Treat group, no significant difference was found (due to many subjects not having working mobile phones/not receiving
Phone access and knowledge of how to use text message was a factor. Inaccurate patient contact information had substantial impact on reminder outreach.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
62
group received 2 texts (7 days before and 1 day before) plus phone call, Control group received only phone call 2 days before.
messages). In the matched control analysis, intervention had 32% reduction in missed appointments.
Robotham, D., Satkunanathan, S., Reynolds, J., Stahl, D., Wykes, T. (2016). Using digital notifications to improve attendance in clinic: Systematic review and meta-analysis. British Medical Journal Open, doi: 10.11136/bmjopen-2016-012116
Level Ib: Experimental Design. Systematic review of RCT and other study designs
Search of MEDLINE, EMBASE, PsycINFO, Cochrane Library. 26 articles were included. Text Intervention. N= 8,345 Control N= 7,731.
Primary outcome was appointment adherence, meta-analysis determined pooled effect size for intervention and control, 95% CI.
Intervention group was 23% more likely to attend appointments and 25% less likely to no-show. RR: 1.23 & 0.75. Two or more notifications increased attendance by 19%, reminders increased cancellations.
Findings for mental health clinics had same outcomes.
Shah, et al., (2016). Targeted reminder phone calls to patients at high-risk of no-show for primary care appointment: A randomized trial. Journal of General Internal Medicine, 31(12), 1460-1466.
Level Ic: Experimental Design Randomized Controlled Trial
2,247 patients determined at risk for appointment no-show randomized into intervention (N=1,129) or control (N=
Between group rates calculated using Absolute Risk difference. T-tests used to determine reimbursement differences between two groups, 95% CI.
No-show rate in intervention arm (22%) was significantly lower than control group (29%). Arrival, cancelation and reschedule rates
Two reminders may be better than one. Only telephone reminders, no SMS.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
63
1,118). Primary outcome was no-show rate. Intervention group received telephone reminder 7 days prior to appointment. Both groups receive reminder call 3 days prior to appointment.
were higher in intervention though not statistically significant.
Teo, A., Forsberg, C., Marsh, H., Saha, S., Dobscha, S. (2017). No-show rates when phone appointment reminders are not directly delivered. Psychiatric Services, 68(11), 1098-1100.
Level IIId: Observational-Analytic Design. Case Controlled Study
Conducted within Dept. of Veteran’s Affairs, participants screened positive for depression. All participants received reminder call Friday before appointment and day before appointment. Type of contact (live, message, no-answer) was logged. N=250
Chi-square analysis used to compare reminder call and appointment attendance.
Live reminders had significantly lower no-show rate (3%) than message (24%) and no reminder (39%). Difference between message and no reminder also significant.
Notes text is becoming more popular, a two-way system that acknowledges receipt may be useful
EVALUATION OF APPOINTMENT REMINDER SYSTEM
64
Tofighi, B., Grazioli, F., Bereket, S., Grossman, E., Aphinyanaphongs, Y., Lee, J. (2017). Text message reminders for improving patient appointment adherence in an office-based Buprenorphine program: A feasibility study. The American Journal on Addictions, 26, 581-586.
Level IIId: Observational Analytic Design. Case Controlled Study
Convenience sample of opiate dependent adults. Non-random, all screened for having a mobile phone, training in using phone was offered, phone numbers were verified. N=93
Text reminders were sent 7,4,1 day prior to appointments. 16-item survey related to technology acceptance was completed afterwards to determine their preferences and opinions. Answers were coded and agreed upon. Show rates were not measured.
91% preferred text message over phone reminders, two was preferred number of reminders, receipt the week prior to appointment was preferred, age was not a factor.
Did not report on adherence/no-show rates.
EVALUATION OF APPOINTMENT REMINDER SYSTEM
65
Appendix B
Interview Protocol
Health Center Evaluation of Appointment Reminder System Process for Psychiatric Appointments
Scheduled within OBGYN/Centering Programs
Meeting Interview Protocol
Part 1: (2:15-2:25)
A. Introduction: Hello. Thank you for coming to this meeting. My name is Lacy Clayton and I am a Doctor of Nursing Practice (DNP) candidate from Drexel University. I am working to identify possible ways to reduce no-show rates for psychiatric appointments scheduled within the OBGYN or Centering clinics. Each of you were identified as someone who has expertise in this area. You will find a copy of the agenda in front of you. This also contains contact information for myself and Judy Faust. Your participation is voluntary and there will be no penalty for not participating. The information received today will be consolidated and synthesized and none of the comments will be attached to any particular person. The aim of this meeting is to gain a complete understanding of how the reminder system is intended to be delivered compared to how it is actually delivered and to get your feedback about challenges and recommendations for improving the appointment reminder system.
B. Please briefly introduce yourself and describe your role, including your specific role in the appointment scheduling and/or reminder process
Part 2: (2:25-3:25)
C. Please describe how the appointment reminder system is intended to be delivered.
a. Probe for: Is there a specific protocol? b. What is the method – phone/text/mail/automated/personal? c. How many reminders are given for one appointment? d. When are the reminder(s) sent? e. Is there a protocol for updating patient contact information?
D. Please describe how the appointment system is actually delivered. a. Probe for: What is the method – phone/text/mail/automated/personal? b. How many reminders are given for one appointment? c. When are the reminder(s) sent? d. How is patient contact information updated?
EVALUATION OF APPOINTMENT REMINDER SYSTEM
66
e. What are the main ways the intended process does not match the delivered process?
i. Why? Part 3: (3:25-3:40)
E. What are the main challenges experienced when trying to implement/deliver the appointment reminders?
a. Probe as needed
F. What are your suggestions for improving the appointment reminder system? a. Probe as needed
Part 4: (3:40-3:55) G: Review findings and confirm/clarify the appointment reminder process. Part 5: (3:55-4:00)
H. Wrap Up and Thank You
EVALUATION OF APPOINTMENT REMINDER SYSTEM
67
Appendix C
Executive Summary
Evaluation of Appointment Reminder System for Perinatal Psychiatric Appointments EXECUTIVE SUMMARY – August, 2018 The high appointment no-show rate for psychiatric appointments scheduled within the Obstetrics Department was identified by staff as a practice area of concern. Patients scheduled for psychiatry appointments within the obstetrics clinic have a no-show rate of 58%. Patients scheduled within the Centering Pregnancy/Parenting program have a no-show rate of 61%. The total number of no-shows from January to August, 2018 is 274. Despite this high no-show rate, psychiatry is the most requested referral within this department. Though missed appointments are experienced across all medical specialties, evidence suggests the rate of psychiatric missed appointments may be twice that of other specialties (Mitchell & Selmes, 2007). Research supports that perinatal women are particularly at risk for missing scheduled appointments due to logistics such as transportation and childcare (Grote et al., 2015). While multiple factors contribute to missed appointments, appointment reminders are proven effective for increasing appointment attendance. An evaluation of the appointment reminder system specific to psychiatry appointments within the obstetrics department was conducted. Five key informants from obstetrics, centering pregnancy, psychiatry, and the call center convened to provide background on the current appointment reminder system. The objective was to determine if the current reminder system aligns with evidence-based practice. Recommendations for improving the appointment reminder system with the goal of increasing appointment attendance are provided below. When perinatal women miss their scheduled psychiatric appointments, they are not accessing the care they need. Depression is the leading cause of disability for persons age 15-44 in the United States (National Institute of Mental Health, 2018). Perinatal women are particularly vulnerable as it relates to mental health with one in seven women receiving a diagnosis of postpartum depression (American Psychiatric Association, 2017) compared to one in 10 women in the general population (Centers for Disease Control, 2017). As few as 35% of pregnant women with a documented history of depression receive treatment. The main predictor for the duration of a postpartum depression episode is the amount of time lapsing before treatment is initiated, making early detection and treatment crucial for these women and their families (Hansotte, Payne & Babich, 2017).
SIGNIFICANCE
BACKGROUND
EVALUATION OF APPOINTMENT REMINDER SYSTEM
68
From a financial perspective, missed appointments are costly. Estimates report that each missed appointment costs an organization $200 (Kheirkhah et al., 2015). Based on this appraisal, missed perinatal psychiatric appointments could potentially cost the obstetrics department over $80,000 annually. Evidence suggests that two appointment reminders within seven days of the appointment are preferred (Robotham et al., 2016). Personal reminders are also shown to improve attendance over automated systems (Teo, Forsberg, Marsh, Saha and Dobscha, 2017). Inaccurate patient contact information is repeatedly found to impede the effectiveness of appointment reminder systems (Percac-Lima et al., 2016). Key findings from the evaluation of the perinatal psychiatry appointment reminder system:
v Patients receive one automated Televox reminder 48 hours before appointment v Centering patients receive additional, personalized reminders from their treatment team v Patient contact information is not routinely updated v The daily Televox report is not being reviewed, resulting in canceled appointments on Televox
report not being noted or offered to waiting patients
The above findings offer opportunity for improvement. The daily Televox report not being reviewed for cancelations is especially noteworthy. This results in available appointment times remaining unfilled, inefficient use of provider time, and an artificial inflation of the no-show rate. Findings from this evaluation offer recommendations for improving the appointment reminder system within the perinatal psychiatric service. Suggested changes have the potential to increase appointment attendance rates thereby increasing patient access to care. The fiscal outcomes of the department may also improve. Appointment attendance rates may be increased by:
v Adding a second Televox reminder within 7 days of the appointment v Implementing a strategy to routinely update patient contact information v Assigning staff to review the daily Televox appointment report v Noting appointment cancelations and scheduling waiting patients into those openings
The institution can employ a rapid cycle improvement strategy to determine if these suggested interventions improve appointment attendance. Scheduling staff can implement the above suggestions over a four-week period, reviewing the no-show rate for that span of time. If the no-show rate is reduced, it may be worthwhile to permanently incorporate these suggestions into the appointment reminder system.
KEY LEARNING
RECOMMENDATIONS
EVALUATION OF APPOINTMENT REMINDER SYSTEM
69
American Psychological Association (2017). What is postpartum depression and anxiety? Retrieved from: http://www.apa.org/pi/women/resources/reports/postpartum- depression.aspx Center for Disease Control (2017). Depression among women. Retrieved from: https://www.cdc.gov/reproductivehealth/depression/index.htm Grote, N., Katon, W., Russo, J., Lohr, M.J., Curran, M., Galvin, E., Carson, K. (2015). Collaborative care for perinatal depression in socioeconomically disadvantaged women: A randomized controlled trial. Depression and Anxiety, 32, 821-834. Hansotte, E., Payne, S., Babich, S. (2017). Positive postpartum depression screening practices and subsequent mental health treatment for low-income women in Western countries: A systematic literature review. Public Health Reviews, 38(3), doi: 10.1186/s40985-017-005 Kheirkhah, P., Feng, Q., Travis, L. M., Tavakoli-Tabasi, S., & Sharafkhaneh, A. (2015). Prevalence, predictors and economic consequences of no-shows. BMC health services research, 16(1), 13. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4714455/pdf/12913_2015_Article_1243.pdf Mitchell, A., Selmes, T. (2007). Why don’t patients attend their appointments? Maintaining engagement with psychiatric services. Advances in Psychiatric Treatment, 13(6), 423-434. National Institute of Mental Health, (2018). Statistics. Retrieved from: https://www.nimh.nih.gov/health/statistics/index.shtml#Intro Percac-Lima, S., Singer, D., Cronin, P., Chang, Y., Zai, A. (2016). Can text messages improve attendance to primary care appointments in underserved populations? Journal of Health Care for the Poor and Underserved, 27(4), 1709-1725. Robotham, D., Satkunanathan, S., Reynolds, J., Stahl, D., Wykes, T. (2016). Using digital notifications to improve attendance in clinic: Systematic review and meta-analysis. British Medical Journal Open, doi: 10.11136/bmjopen-2016-012116 Teo, A., Forsberg, C., Marsh, H., Saha, S., Dobscha, S. (2017). No-show rates when phone appointment reminders are not directly delivered. Psychiatric Services, doi: 10.1176/appi.ps.201700128
REFERENCES