ARTICLE

Integration of PediatricMental Health Care: AnEvidence-Based Workshopfor Primary Care Providers

Stacy D. Burka, DNP, FPMHNP-BC, FNP-BC, Susan N. Van Cleve, DNP, RN,CPNP-PC, PMHS, Sheree Shafer, DNP, FNP-BC, PMHCNS-BC, PMHS,& Jennifer L. Barkin, PhD

ABSTRACTIntroduction: Pediatric primary care providers (PCPs) are be-ing asked to care for children with mental health (MH) disor-ders but cite inadequate training as a barrier. An intensiveworkshop may improve the PCPs� level of knowledge andlead to an increase in quality care for children with MH dis-orders. We compared pediatric PCPs� knowledge, comfort,and practice in the evaluation and management of pediatricpatients with attention deficit–hyperactivity disorder, depres-sion, anxiety, and autism spectrum disorders before and aftera 2-day educational workshop.Method: Study participants (n = 30) were recruited from ruralareas of Pennsylvania. A pre- and posttest design was used. A15-question multiple choice knowledge test and a 19-question survey of comfort and practice were administeredbefore and after the workshop.

StacyD.Burka,RegisteredNurse, InterimHealthCare,Pittsburgh,PA.

Susan N. Van Cleve, Associate Professor, School of Nursing and

Health Sciences, Robert Morris University, Moon Township, PA.

Sheree Shafer, Family Nurse Practitioner and Psychiatric MentalHealth Clinical Nurse Specialist, Children’s Community Care

Armstrong Pediatrics, Kittanning, PA.

Jennifer L. Barkin, Assistant Professor of Community Medicine,Mercer University School of Medicine, Macon, GA.

Conflicts of interest: None to report.

Correspondence: Stacy D. Burka, DNP, FPMHNP-BC, FNP-BC,134 Merry Lane, Butler, PA 16001; e-mail: sdbst14@mail.rmu.edu.

0891-5245/$36.00

CopyrightQ 2014 by theNationalAssociationofPediatricNurse

Practitioners. Published by Elsevier Inc. All rights reserved.

Published online May 6, 2013.

http://dx.doi.org/10.1016/j.pedhc.2012.10.006

www.jpedhc.org

Results: The mean knowledge test number correct increasedfrom 9.19 before the workshop to 12.23 after the workshop(p < .0001). Survey scores increased from 34.6 before theworkshop to 44.14 after the workshop (p < .0001).Discussion: Intensive workshops may be an effective methodof training PCPs on provision of MH care in pediatric primarycare practice. J Pediatr Health Care. (2014) 28, 23-34.

KEY WORDSMental health integration, primary care, pediatric mentalhealth, workshop

In the United States, as many as one out of every fourto five youth will experience a mental health (MH)problem over the course of their lifetime (Merikangaset al., 2010). Access to specialty MH care is problematicbecause of a shortage of child and adolescent psychia-trists, geographic unavailability of mental health ser-vices (United States Department of Health andHuman Services [USDHHS] Health Resources andServices Administration, 2012), and lengthy delays be-tween referral and intake (Heneghan et al., 2008). Themost common disorders evaluated in primary caresettings include attention deficit–hyperactivity disorder(ADHD), depression, and anxiety (American Academyof Child and Adolescent Psychiatry [AACAP] WorkGroup onQuality Issues, 2007a, 2007b, 2007c). Increas-ing numbers of childrenwith autism are also being eval-uated and followed up in primary care (Golnik &Maccabee-Ryaboy, 2010).Pediatric primary care providers (PCPs), including

nurse practitioners (NPs), physician assistants (PAs),and physicians, are being asked to care for children

January/February 2014 23

with behavioral and MH disorders in their practices butare inadequately prepared to care for these patients(Kolko, Campo, Kelleher, & Cheng 2010). PCPs self-report unease and inadequate training as barriers to ini-tiating care (Carbone, Behl, Azor, & Murphy, 2010;Kolko et al., 2010). Additional identifiedbarriers includelack of time to thoroughly evaluate the pediatric patientwith mental health concerns (AACAP Committee onHealth Care Access and Economics Task Force onMental Health, 2009; Kolko et al., 2010; Schlesinger,2008) and lack of reimbursement to PCPs for mentalhealth treatment (Barclay, 2009; Heneghan et al., 2008;National Institute for Health Care Management, 2009).

Further training in the diagnosis and management ofcommonMHdisorders and aplan for integrationof caremay improve the PCPs� level of comfort, lead to an in-crease in high-quality care provided in the primarycare setting, and help fill the void that exists in accessto and provision of pediatric mental health care in theUnited States today.

NEED FOR INTEGRATION OF MENTAL HEALTHINTO PRIMARY CAREThe need for integration of pediatric mental health ser-vices into pediatric primary care is great. Children andadolescents are being seen in greater numbers in pri-mary care offices for the assessment, evaluation, andtreatment of disorders including ADHD, anxiety, anddepression (AACAP Work Group on Quality Issues,2007a, 2007b, 2007c). According to the most recentCenters for Disease Control and Prevention (CDC) re-port, the diagnosis of autism spectrum disorder is on

With increasingnumbers ofchildren andadolescentsneeding ongoingmental andbehavioral healthcare, it is vital thatPCPs meet theneed and thatpractices developplans to provideintegrated mentalhealth care forpatients.

the rise, with an aver-age of 1 out of every88 children in theUnited States havingthis diagnosis (CDC,2012). With increasingnumbers of childrenand adolescents need-ing ongoing mentaland behavioral healthcare, it is vital thatPCPs meet the needand that practices de-velop plans to provideintegrated mentalhealth care for patients.In an integratedmodel,PCPs may be able tomanage the majorityof mild to moderatedisorders in their prac-tices, while referring

the more complex and serious disorders to psychia-trists, psychiatric nurse practitioners, or mental healthspecialists.

24 Volume 28 � Number 1

Fifteen million children and adolescents have beenidentified as needing a psychiatrist (NationalAssociation of Pediatric Nurse Practitioners [NAPNAP],2007), and yet approximately 65% of these childrenand adolescents receiveminimal or no services for theirMH need (Merikangas et al., 2011). According toAACAP (2010), approximately 7000 child and adoles-cent psychiatrists are available in the United States,with about 300 per year completing training (Barclay,2009). Geographic distribution varies, and some ruralareas have virtually no access to care (Thomas, Ellis,Konrad, Holzer, & Morissey, 2009). For every 100,000youth in the United States, there is an average of 8.7child and adolescent psychiatrists available, with widevariations between states (Thomas & Holzer, 2006).This lack of services has major implications for childrenand families. Many children go untreated, and for thesechildren, the consequences are serious and farreaching.Children and adolescents with untreated MH disor-

ders are at increased risk for a myriad of psychosocialproblems that affect the child or adolescent and familyin the present and future. Children and adolescentswith aMHdisorder may experience impairment in rela-tionships with family, peers, and within the school sys-tem (Subcommittee on Attention-Deficit/HyperactivityDisorder, Steering Committee on Quality Improvementand Management, 2011). The child or adolescent withaMH disorder may have problems interacting with par-ents, siblings, or both, thus creating a strained and po-tentially dysfunctional family dynamic (Foley, 2011).They may have difficulties with forming peer relation-ships and/or functioning effectively in the schoolenvironment, leading to academic failure, poor em-ployment opportunities, and poverty (AACAP WorkGroup on Quality Issues, 2007b; Barclay, 2009). Finan-cially, this burden is substantial. Loss of income alone inpersons with ADHD is estimated to be at a minimum$67 billion (Biederman & Farone, 2006). In addition,adolescents may demonstrate an increase in risk-taking behaviors, resulting in teen pregnancy and legalproblems (AACAP Work Group on Quality Issues,2007b; Barclay, 2009). Children and adolescents withan untreated MH disorder are also at risk for additionalcomorbidities, substance abuse (AACAP Work Groupon Quality Issues, 2007b; Barclay, 2009), injuries/acci-dents (AACAPADHDResource Center, 2010), andmor-tality and suicide. According to AACAP (2008), suicideis the third leading cause of death in 15- to 24-year-olds and ranks sixth in those in the 5- to 14-year agegroup, thus demonstrating the need for appropriateand timely treatment.

CURRENT INITIATIVESIn response to the abject need to address this caredeficit in a vulnerable population, the American Acad-emy of Pediatrics (AAP) has described the ‘‘medical

Journal of Pediatric Health Care

home’’ as a means by which to provide accessible,family centered, coordinated, comprehensive, contin-uous, compassionate, and culturally effective care forall children, including those with MH disorders andother chronic conditions (AAP, 2002). HealthyPeople 2020 also identifies improvement of MHthrough prevention and access to appropriate qualityMH services as a goal for 2020 (USDHHS, 2010). TheUnited States Preventive Services Task Force(USPSTF) recommends that MH screening be incorpo-rated into routine primary care for adolescents(USPSTF, 2009). There is a clear call for PCPs to pro-vide MH care to children and adolescents in theirpractices now and in the future.

BENEFITS OF INTEGRATION OF MENTALHEALTH SERVICES INTO PRIMARY CAREAccess to care is a problem for pediatric MH patientsand their families. For many children, the PCP may bethe point of access to MH care. For persons in ruraland urban areas, child psychiatrists and/or othermentalhealth services may not be readily available, but theyare willing to go to their pediatric practice for MH ser-vices (Connor et al., 2006). Geographic inaccessibilityfor MH care by psychiatrists and other MH providerscauses lost income due to parents missing work, in-creased transportation costs, missed school time forthe child or adolescent, and decreased parental timefor other children. Even when specialty psychiatriccare is more readily available, there are often lengthydelays in scheduling appointments (Heneghan et al.,2008).

In addition to improved accessibility, pediatric pa-tients and their families may be more willing to utilizeMH services offered within primary care settings be-cause of the existing relationship between the childand provider (Foy, Kelleher, & Laraque, 2010). The stig-matization of MH providers and services has been pro-posed as a barrier to the utilization of MH services(Smith, 2008), and accessing MH services through pri-mary care may serve as a destigmatizing factor (Foyet al., 2010). Providing primarymedical care in conjunc-tion with MH care facilitates treatment of the patient asa whole person.

BARRIERS TO PROVIDING MENTAL HEALTHSERVICES IN PRIMARY CAREIt is evident that early identification and subsequentmanagement of MH disorders is beneficial to pediatricpatients and their families. PCPs are in auniquepositionto assess, diagnose, and treat the patient, coordinatecare, and evaluate outcomes on an ongoing basis(Kelleher, Campo, & Gardner, 2006). Unfortunately,provider unease and lack of training related to diagnos-ing and managing these patients may limit the PCP�sefficacy in this role.

www.jpedhc.org

PCPs are hesitant to initiate medications in childrenwith MH disorders (Carbone et al., 2010), which maybe due to black box warnings (Hassanin et al., 2010;Lovrin, 2009). In 2004, the United States Food andDrug Administration (FDA) issued a black box warningabout an increased risk of suicidal thoughts or behaviorin children and adolescents being treatedwith selectiveserotonin reuptake inhibitor (SSRI) antidepressantmedications. Although the evidence is clear that thebenefits of antidepressant medications outweigh therisks (Bridge et al., 2007) and that evidence-based prac-tice in psychopharmacology supports the safe and ef-fective use of psychotropic medications (AACAPWork Group on Quality Issues, 2009), providers con-tinue to be reluctant to prescribe these medications,perhaps because of the issuance of black box warnings(Campo, 2009; Hassanin et al., 2010). PCPs also expressreservations regarding the evaluation andmanagementof other common pediatric MH disorders. PCPs hold animportant role in themanagement of autism in children(Carbone, Farley, & Davis, 2010), yet they report a lackof skill in providing a medical home to these children(Golnik, Ireland, & Borowsky, 2009).In addition to lack of knowledge and comfort in iden-

tifying andmanaging children and adolescentswithMHdisorders, time constraints and reimbursement issuesare problematic. According to Kolko et al. (2010),PCPs identify time pressures as a serious barrier tocare. A thorough initial psychiatric interview may takeup to 60 minutes, with follow-up appointments takingan average of 20 to 30 minutes. It is recommendedthat children and adolescents be interviewed togetherwith their parents and also separately (AACAPCommittee on Health Care Access and EconomicsTask Force on Mental Health, 2009), therefore addingto the length of the appointment. Schlesinger (2008)cites the time required to perform a traditional psychiat-ric interview as a barrier to delivery of MH services inprimary care.Disparity in reimbursement also affects PCPs� will-

ingness and ability to provide services in primarycare. Differences between public and private healthplans create provider insecurity regarding paymentfor the provision of MH services. Reimbursement con-cerns include behavioral health ‘‘carve outs’’ in whicha managed behavioral health care organization paysonly to the contracted behavioral health specialistson its ‘‘panel’’ for these services. Pediatricians typi-cally lack credentialing to be on the panel and aretherefore considered ineligible to bill for these ser-vices (Barclay, 2009; National Institute for HealthCare Management, 2009). Even when PCPs can billfor care delivered, they cite lack of payment fortime spent on consults and in telephone communica-tion as a significant barrier to care (AACAP Committeeon Health Care Access and Economics Task Force onMental Health, 2009; Barclay, 2009).

January/February 2014 25

EDUCATION OF PRIMARY CARE PROVIDERSPCPs are in need of education and training on how toprovide MH care in primary care settings, but few tar-geted programs are available to educate PCPs on howto provide this care and how to make changes to prac-tice. Several organizations offer resources for providersinterested in integration of MH care. The REACH Insti-tute provides programs for PCPs on psychopharmacol-ogy and child mental health training for PCPs (TheREACH Institute, 2010). The National Alliance on Men-tal Illness (NAMI) and TeenScreen National Center atColumbia University provide resources such as screen-ing tools, webinars, and outlines with steps to guide in-tegration of care (NAMI, 2011; TeenScreen NationalCenter for Mental Health Checkups at ColumbiaUniversity, 2003). The KySS Guide to Child and Adoles-cent Mental Health Screening, Early Intervention, andHealth Promotion (Melnyk and Moldenhauer, 2006) isdesigned specifically for PCPs interested in integratingMH into pediatric primary care. The KySS Guide pro-vides information on MH diagnoses with correspond-ing Diagnostic and Statistical Manual of MentalDisorders (DSM-IV) criteria. A new edition of this guideis due to be released in 2013. Practice parameters focus-

Educatingproviders on bestpractices for theuse of MHscreening tools,assessment,diagnosis, andtreatment ofchildren withbehavioral and MHdisorders iscritically important.

ing on diagnoses suchas ADHD, depression,anxiety, and prescrib-ing of psychotropicmedications are aimedtoward PCPs and areavailable from AACAP(AACAP Work Groupon Quality Issues,2007a-c, 2009).

Educating providerson best practices forthe use of MH screen-ing tools, assessment,diagnosis, and treat-ment of children withbehavioral andMHdis-orders is critically im-portant. An optimal

method of teaching this complex information may bean intensive workshop (Price, 2010). A workshop isan appropriate venue when ‘‘skills such as listening,conflictmanagement, history taking and priority settingare required’’ (Price, 2010). In this setting, participantshave the opportunity to learn from the workshop facil-itator as well as through communication with otherswho have similar goals (Horsfall & Cleary, 2008).

PURPOSEThe purpose of this project was to provide an intensiveworkshop to PCPs from Pennsylvania on how to inte-grate and provide behavioral and MH care to childrenand adolescents in their practices. Two 2-day intensive

26 Volume 28 � Number 1

workshops were designed to present information onthe assessment, diagnosis, and treatment of childrenwith ADHD and mild to moderate anxiety and depres-sion and how to provide primary care to children withautism spectrum disorder. A session on tools for inte-grating MH into primary care was also provided. Weevaluated the effect of these educational workshopsand examined the PCPs� level of knowledge beforeand after the workshop.We also examined the changesin PCP practice regarding MH 1 month after the work-shop. We chose a workshop format because we be-lieved that the reciprocal interaction between thefacilitators and the participants would be an ideal wayto ascertain their level of understanding and ongoinglearning needs.

METHODOLOGYDesignThe studyused a pretest/posttest design to compare thevariables of level of knowledge and level of comfort/practice before and after a 2-day workshop. A test(Figure) of PCP knowledge regarding the assessment,diagnosis, and treatment of ADHD, anxiety, depres-sion, and autism spectrum disorders was administeredbefore theworkshop began onday 1 and at the comple-tion of day 2. The test (Figure) was created by the au-thors specifically for the study and was reviewed byexperts for content validity. A survey (Table 1) was de-veloped by the authors to assess practice and comfortlevels related to evaluation and management of pediat-ric MH patients. The survey (Table 1) was administeredbefore the start of presentations on day 1 and again at 1month after the workshop. The survey (Table 1) in-cluded questions on use of screening tools, utilizationof pharmacologic and nonpharmacologic measures,and level of comfort in the evaluation andmanagementof common pediatric mental health disorders, alongwith questions related to coding and billing for mentalhealth services. The survey was reviewed and critiquedby clinical experts prior to administration for contentvalidity.

SampleParticipants were recruited, with permission, from thePennsylvania AAP Medical Home Initiative Practices,focusing on those from central, northeastern, andnorthwesternPennsylvania, aswell as from threeNPor-ganizations in Pennsylvania. Thirty pediatric PCPs self-identified as a convenience sample to attend one of twoworkshops. Through ongoing collaboration with theAAP, registration was opened to members of the Medi-cal Home Initiative and to other PCPs. Inclusion criteriawere pediatric NPs, family NPs, pediatricians, familypractice physicians, PAs, RNs, and social workers whopracticed at least half of the time in pediatric primarycare. Continuing education credits were provided bythe AAP andNAPNAP. Therewas no cost to participants

Journal of Pediatric Health Care

FIGURE. Pretest/Posttest of Knowledge (page 1).

Research Number _____________

Please circle the answer that is correct. There is only one correct answer per question.

1. According to evidence-based practice, adolescent depression is best treated by:

A. Six weeks of supportive therapy.

B. Medication only.

C. Watching and waiting.

D. Combination of therapy and medication.

2. You are providing education on medication for depression. Part of this education includes:

A. Fluoxetine and Lexapro are the only antidepressants that have FDA approval for treatment of adolescents.

B. If you see no response with one antidepressant, there is no reason to try another.

C. The FDA placed a Black Box Warning on antidepressants for use in adolescents, stating that these medications increase suicide

D. Through research we know that very few adolescents respond to antidepressant medication.

3. The best resource to use to make a diagnosis of depression is:

A. An evidence-based depression screening tool.

B. Evidence-based guidelines on depression.

C. An assessment from a local mental health worker.

D. The Diagnostic and Statistical Manual (DSM-IV).

4. Generalized anxiety disorder is characterized by:

A. Inability to separate from parents.

B. Severe worries at least once weekly.

C. Inability to control worries.

D. Always accompanied by irritability and sadness.

5. An example of evidence-based treatment for anxiety is:

A. Avoidance of the anxiety-provoking stimulus and cognitive behavioral therapy.

B. Flooding with the anxiety-provoking stimulus and cognitive behavioral therapy.

C. Medication management with narcotics and cognitive behavioral therapy.

D. Medication management with narcotics and flooding with the anxiety-provoking stimulus.

6. Medications indicated for the first-line treatment of anxiety in adolescents include:

A. Narcotics. B. SSRIs. C. Anxiolytics. D. None

7. Characteristics of attention deficit–hyperactivity disorder include:

A. Hyperactivity, tics, short attention span, and distractibility.

B. Presence of symptoms of ADHD exclusively in the school setting.

C. Hyperactivity, impulsivity, distractibility, and a short attention span.

D. Short attention span, distractibility, panic attacks, and sleeplessness.

www.jpedhc.org January/February 2014 27

FIGURE. (Continued).

8. The best way to describe the action of stimulant medication is:

A. Stimulants sedate an active child.

B. Stimulants are the missing chemicals in the brain.

C. Stimulants enhance serotonin levels.

D. Stimulants decrease the breakdown of certain neurotransmitters.

9. The best description of Atomoxetine is:

A. This medication is both an antidepressant and a stimulant.

B. This is a medication that can be taken as needed.

C. This is a medication that requires 2 weeks for full effect.

D. This is a medication that is a nonstimulant.

10. In a patient with no risk factors, the American Academy of Pediatrics recommends that a standardized autism specific screening tool be used:

A. Not at all.

B. At regular intervals (i.e. 9, 18, and 24 months).

C. Beginning every 6 months at the 24-month visit.

D. Only when the parent or provider notes a behavioral concern.

11. Testing that should be considered for all young children with autism spectrum disorder is:

A. Audiology testing.

B. EEG.

C. MRI.

D. Laboratory work for metabolic studies.

12. Parents of a child who is newly diagnosed with autism express confusion related to their child’s treatment regimen. Which of the following would you tell them?

A. Behavioral modification strategies are the only appropriate intervention when treating autism.

B. Medications are ineffective in treating symptoms of autism because the exact cause of autism is unknown. C. Although no medications can improve the core signs of autism, certain medications can help control symptoms of behaviors that may accompany autism.

D. When treating autism, it is always necessary to incorporate both pharmacologic and nonpharmacologic modalities.

13. You are seeing a 10-year-old girl who meets criteria for ADHD. You code this as:

A. Hyperkinicity.

B. Behavioral disorder NOS.

C. ADHD

D. Disruptive behavioral disorder.

14. You are seeing a 14-year-old teen for an evaluation and treatment for depression and anxiety. He mentions that he has been irritable and is having difficulty sleeping. You spend 60 minutes with him. This may be billed:

A. According to level of complexity.

B. As counseling time.

C. According to number of symptoms.

D. As evaluation of sleep disorder.

15. Screening for anxiety might include use of what tool?

A. Vanderbilt

B. Conners’ parent

C. M CHAT

D. SCARED

28 Volume 28 � Number 1 Journal of Pediatric Health Care

TABLE 1. Presurvey and postsurvey of comfort and practice

NeverVeryrarely Rarely Occasionally Frequently

Veryfrequently

1. I am using screening tools for the evaluation of ADHD. 0 1 2 3 4 52. I am prescribing medications for the treatment of ADHD. 0 1 2 3 4 53. I am using nonpharmacologic measures for the treatment of ADHD. 0 1 2 3 4 54. I am comfortable with the evaluation and management of ADHD. 0 1 2 3 4 55. I am using screening tools for the evaluation of depression. 0 1 2 3 4 56. I am prescribing medications for the treatment of depression. 0 1 2 3 4 57. I am using nonpharmacologic measures for the treatment of

depression.0 1 2 3 4 5

8. I am comfortable with the evaluation and management ofdepression.

0 1 2 3 4 5

9. I am using screening tools for the evaluation of anxiety. 0 1 2 3 4 510. I am prescribing medications for the treatment of anxiety. 0 1 2 3 4 511. I am using nonpharmacologicmeasures for the treatment of anxiety. 0 1 2 3 4 512. I am comfortable with the evaluation and management of anxiety. 0 1 2 3 4 513. I am using screening tools for the evaluation of ASD. 0 1 2 3 4 514. I am prescribing medications for treatment of symptoms of ASD. 0 1 2 3 4 515. I am using nonpharmacologic measures for the treatment of

symptoms of ASD.0 1 2 3 4 5

16. I am comfortable with the evaluation and management of ASD. 0 1 2 3 4 517. I am accurately coding and billing for mental health disorders in my

practice.0 1 2 3 4 5

18. I am comfortable with my knowledge of coding and billing for mentalhealth disorders in my practice.

0 1 2 3 4 5

19. Our practice is getting reimbursed appropriately for coding andbilling for mental health disorders.

0 1 2 3 4 5

ADHD, attention deficit–hyperactivity disorder; ASD, autism spectrum disorders.

Directions: Please circle the answer that most closely describes your current level of comfort and/or practice.

to attend either workshop. This project was funded bythe Jewish Health Care Foundation, Pittsburgh, PA.

Setting/InterventionWorkshops targeted but were not limited to pediatricPCPs from rural counties in central, northeastern, andnorthwestern Pennsylvania. Two separate 2-day edu-cational workshops were held in Clarion and Nanti-coke, Pennsylvania, which both are designated asmedically underserved counties (USDHHS, 2012).

Workshop content included the evaluation andman-agement of ADHD, autism, and mild to moderate de-pression and anxiety, as well as diagnostic criteria foreach disorder. Review of administration, scoring, andinterpretation of multiple screening tools was includedin theworkshop content. These tools included ConnersRating Scales, Vanderbilt assessments, the ModifiedChecklist for Autism in Toddlers (M-CHAT), the Screenfor Child Anxiety Related Disorders (SCARED), Moodand Feelings Questionnaire, Columbia DISC Depres-sion Scale, the CRAFFT screen, and the Patient HealthQuestionnaire (PHQ-9). Pharmacologic treatmentswere discussed with emphasis on appropriate dosing,common adverse effects, contraindications, and anytesting or laboratory work to be done prior to initiationor continuation of a medication. Nonpharmacologicmanagement was also discussed with a focus on briefpsychotherapeutic interventions that could be per-

www.jpedhc.org

formed in a primary care setting. Practice with moodmonitoring and problem solving as well as with relaxa-tion techniques, such as deep breathing and guided im-agery, was included. Coding, billing, documentation,reimbursement, and recommendations on how to inte-grate mental health care into a primary care practicewere discussed.Ahybridmethodof teaching strategywas chosen, en-

compassing both a lecture format and presentation anddiscussion of case studies. Case studies were presentedby the speakers, and workshop attendees were invitedto expand upon and problem solve these cases and toshare their own experiences. Research has demon-strated that problem-based learning is an effectivestrategy for adult learners, particularly those in healthcare–related professions (Chikotas, 2008). Problem-based learning is ‘‘a method of group learning thatuses true-to-life problems as a stimulus to developproblem-solving skills and acquire domain knowledge(nursing)’’ (Chikotas, 2008, p. 360). Adult learners haveexperiences thatmay act as impetus for their own learn-ing as well as other group members� learning, and thesharing of these experiences may aid in identifyinglearning needs. Course content was developed and de-livered by the investigators, who are doctorally pre-pared nurse practitioners with expertise in the area ofbehavioral andmental health andwork in pediatric pri-mary care practices as mental health subspecialists. All

January/February 2014 29

content was reviewed and critiqued by clinical expertsprior to the workshops for content validity, and thesame content was presented at both workshops. Partic-ipants were given manuals and access to informationonline.

InstrumentsA 15-question multiple choice test (Figure) was utilizedto assess participants� level of knowledge. Each ques-tion had one possible correct answer. A 19-questionsurvey (Table 1) was used to assess level of comfortand practice. Each question could be answered witha response ranging from 0 to 5 using a Likert scale to in-dicate participants� level of comfort and practice. Twoopen-ended questions were included to allow partici-pants the opportunity to identify, in their own words,perceived challenges to incorporation of care and inter-ventions they thought would be helpful in facilitatingintegration of MH care delivery into their practice. Inaddition, participants were asked to complete an eval-uation of the workshop, including the venue, thespeakers, and content of the workshop. The evaluationinstrumentswere developedby the investigators for thepurposes of this study.

Data CollectionInstitutional Review Board approval for this study wasobtained through Robert Morris University InstitutionalReview Board Committee. Potential participants wereinformed of the date, time, and location of the work-shop. At the beginning of workshop day 1, the investi-gators reviewed the informed consent and addressedall questions and concerns. Participants who agreedto take part in the research study were asked to signthe consent. Each participant was assigned a researchnumber, which was provided in a sealed envelope, toensure confidentiality. They were asked to place thisnumber at the top of each page of all tests and surveys.Demographic information was collected. A test ofknowledge (Figure) and a survey of practice (Table 1)were given at the start of day 1. Both workshop dayslasted approximately 8 hours. The knowledge test(Figure) was readministered to participants immedi-ately at the end of day 2. The practice survey(Table 1)was sent electronically to participants 1monthafter the workshop. One month between practice sur-veys gave the participants time to begin effecting prac-tice change. If no responsewas received, awritten copyof the practice survey was sent via U.S. mail with an en-velope enclosed for return of the survey. An additionale-mail reminder with an electronic copy of the surveywas sent an additionalweek after that if therewas no re-sponse.

Data AnalysisDescriptive statistics were used to describe demo-graphic data (Table 2). The mean and standard devia-

30 Volume 28 � Number 1

tion were calculated for each of the continuousvariables, including age of the participant, number ofyears in practice, and number of years spent in primarycare. Percentages were calculated to quantify thecomponents of the categorical variables. Categoricalvariables includedgender, if children/adolescents com-pose 50% of the population seen, work status, cliniciantype, and highest degree attained.Pretests and posttests of knowledge were scored ac-

cording to the number correct out of a possible totalscore range of 0 to 15,with 15 indicating a perfect score.Mean scores were calculated for the surveys of comfortand practice completed before and after the workshopusing a Likert scale and summing across all itemsfor a possible total score range of 0 to 95. A higher scoreindicated a higher level of comfort and practice. Pairedt-tests were conducted to assess pre- and postwork-shop differences for both provider knowledge andpractice.

RESULTSThe mean age of study participants was 47.54 years,with 17.97 years in practice and 10.84 of those yearswithin primary care. The sample (N = 30) consisted en-tirely of women. The majority of participants (73.33%)indicated that adolescents/children constituted at least50% of their practice, whereas five participants(16.67%) said that adolescents/children did not consti-tute at least 50% of their practice. Most participants(80%) worked full time. The majority practiced as NPs(45.71%), with other clinician types represented as fol-lows: RN (5.71%), physician (5.71%), and PA (5.71%).More than 80% of participants had a master�s degree,two (6.67%) had a doctoral degree, and two (6.67%)had a medical degree. One participant had a bachelor�sdegree andonehad an associate�s degree, each at 3.33%(Table 2).The difference between knowledge tests taken be-

fore and after the workshop was calculated; the testsof 26 participants (86.7%) were usable, with tests offour participants (13.3%) designated as missing be-cause of nonresponse to one or more knowledge testquestions. The mean knowledge test number correctbefore the workshop was 9.19 (SD = 1.833), witha mean of 12.23 questions correct after the workshop(SD = 1.505). This result was statistically significant(p < .0001; Table 3).The difference between pre- and postsurveys of

comfort and practice was calculated; the surveys of 20participants (66.7%) were usable, with surveys of 10participants designated as missing because of nonre-sponse to one or more survey questions and/or failureto return thepostsurvey. Thepresurveymean scorewas34.6 (SD = 21.564), with a postsurvey score of 44.15(SD = 22.797). This result was also statistically signifi-cant (p < .0001; Table 3).

Journal of Pediatric Health Care

TABLE 2. Workshop participantcharacteristics

Variables N Mean (SD) %

ContinuousAge (years) 28 47.54 (12.90)Years of practice 25 17.97 (14.11)Years in primary care 20 10.84 (9.87)

CategoricalGender

Male 0 0Female 30 100

Adolescents/children constitute50% of practice

Yes 22 73.33No 5 16.67Missing 3 10

Work statusFull time 24 80Part time 6 20

Clinician typeNP 16 45.71RN 2 5.71MD 2 5.71PA 2 5.71Other 11 31.43Missing 2 5.71

Highest degree attainedAssociate 1 3.33Bachelor 1 3.33Master 24 80Doctorate 2 6.67Medical degree 2 6.67

MD, medical doctor; NP, nurse practitioner; PA, physician�s as-

sistant; RN, registered nurse.

DISCUSSIONThe workshop implemented in this study appears tohave achieved its purpose of increasing PCP knowl-edge and comfort in relation to the evaluation andman-agement of common pediatric MH disorders ina primary care setting. Knowledge level of PCPs wassignificantly increased after a 2-day workshop, and sur-veys demonstrated an increase in comfort level anda change in practice 1 month after the workshop. Theworkshop format itself was also well received. In re-sponse to open-ended questions and to evaluations,participants reflected on aspects of the workshop thatthey most enjoyed. Comments included:

TABLE 3. Results of paired samples test

Before the workshop

Mean SD

Knowledge test No. correct 9.19 1.833Survey of comfort/practice 34.6 21.564

The posttest of knowledge was administered at the end of the workshop.

after the workshop. Paired t-tests were performed before and after the w

www.jpedhc.org

‘‘Excellent. Exactly what I was looking to take back tomy practice.’’‘‘Exactly what I wanted to enhance my knowledge

base and skills in my private practice primary care set-ting.’’‘‘I really liked the face to face presentation versus on-

line.’’‘‘The small group size—ability to ask questions dur-

ing presentations.’’‘‘Speakers availed themselves to collegial discussion

and case studies. I also enjoyed the peer interaction fa-cilitated by the speakers.’’‘‘Opportunity to network/share ideas with col-

leagues in my field.’’

Participant comments on workshop improvementsuggested that future inclusion of content on eatingdisorders, bullying, and posttraumatic stress disorderwould be of benefit. Participants also identified useand scoring of screening tools, delivery of brief cogni-tive behavioral therapy interventions, and discussionof sexual behaviors with adolescents as areas inwhich they desired additional time for practical appli-cation.Although the significance of our results indicates

that the workshop format is effective in increasingprovider knowledge and comfort, additional obsta-cles to the integration of MH care exist. There are ad-ditional challenges that are inherent to the rollout ofan educational program on a widespread basis. Iden-tification of experts in the field to act as workshopleaders will be a critical first step in expanding theworkshop to other areas of the United States. The fi-nancial costs associated with registration fees andwith time spent away from practice to participate inthe workshop may discourage PCPs from attending.The upfront costs associated with purchase of screen-ing tools and for utilization of office staff to supportMH integration efforts may be prohibitive for somepractices. Office staff may require further training inthe aspects of MH integration that are unique to theirrole, such as administration and scoring of screeningtools, triaging psychiatric emergencies, and schedul-ing of appointments. Some practices may also desireinitial onsite program support of integration efforts,which our program did not provide.Offering of this type of program via the Internet

may aid in decreasing some of these obstacles.

After the workshop

p valueMean SD

12.23 1.505 < .000144.15 22.797 < .0001

The postsurvey of comfort and practice was administered 1 month

orkshop for the knowledge test and the survey of comfort/practice.

January/February 2014 31

Implementation ofthese workshopsmay aid ineducating PCPsabout best practicein the provision ofcare for youth withbehavioral and MHdisorders.

Provider choice of an online or on-site venue offersflexibility to busy professionals and may increaseworkshop participation because the PCP would nothave to adjust his or her schedule or take time awayfrom practice to be able to attend a workshop onsite. A webinar delivers a degree of flexibility whilealso allowing participants to ask questions and inter-act with others in real time, a facet of the on-siteworkshop that was well received. In addition, partic-ipation in Web-based formats may allow PCPs to cus-tomize their learning by allowing them to focus onareas identified as most important to their practice.Prior research has demonstrated that online learningis a desired, effective, and cost-efficient way to pro-vide continuing education to professionals(Bromley, 2010; Church et al., 2010). An evidence-based concept called spaced education (SE) hasbeen found to be of benefit in the enhancement ofprior face-to-face learning (Kerfoot et al., 2010;Matzie, Kerfoot, Hafler, & Breen, 2009) SE is describedas a process in which ‘‘participants� receive shortmultiple-choice questions and feedback via e-mail ina repeating pattern over a number of weeks’’ (Shaw,Long, Chopra, & Kerfoot, 2011). Utilization of the SEconcept at designated intervals after a more tradi-tional face-to-face encounter, such as our workshop,may aid in continued retention of knowledge andpractice change over time.

Study strengths included content presentation bypractitionerswith expertise in the treatment of pediatricmental illness, whichmay explain participants� positivereaction to theworkshop. Both presenting practitionersalso hold faculty positions, and their teaching experi-ence played an important part in conveying the infor-mation in an engaging and effective manner. Theknowledge test and comfort surveys were examinedfor content validity and ease of administration by pro-fessionalswith expertise in survey/instrument develop-ment. To allow for each participant to choose herpreferred method of response to the follow-up survey,surveys were sent electronically as well as via U.S. mail.Weekly reminder e-mail messages were sent if a re-sponse was not received. These factors, in combina-tion, likely contributed to the responsiveness of theparticipants and the success of the workshop asa whole.

Limitations of the study included the small number ofparticipants (n = 30) and the single gender sample (allfemale),which limit generalizability of findings. Conve-nience sampling in which participants self-identified asdesiring increased knowledge was used. Sampling ofless invested participants may result in different find-ings. Additionally, the participants were primarilyfrom rural practices in Pennsylvania, and differencesmay be found in urban areas and/or in other states.There was also a narrow time frame between adminis-tration of the pretest and posttest. Although this time

32 Volume 28 � Number 1

frame was helpful in establishing the workshop as thelikely cause of providers� increased knowledge andcomfort level, it is not advantageous to the assessmentof information retention.

CONCLUSIONSMany factors prevent PCPs from providing behavioraland MH care to children and adolescents in primarycare settings. Lack of education and training has beenidentified as a barrier. This study indicates that an inten-sive workshop aimed at increasing PCP knowledge andcomfort may be an effective solution. In accordancewith participants� suggestions, inclusion of additionaltime for role-play scenarios and practice of case studiesin future workshops may be beneficial. Offering ofbreakout sessions would allow for participants to indi-vidualize their learning by choosing to engage in ses-sions most pertinent to their practice. Future

researchers may wantto attempt to replicateour study in variedpopulations and usea more prolongedtime frame betweenpretesting andposttest-ing to assess for ongo-ing retention ofinformation. It is a pos-sibility that future re-search could continueto follow the partici-pants from our study to assess for continuing and sustained practice changeover time. Should findings from additional studies con-curwith the results of this study, the intensiveworkshopformat should be considered for use on a broader scale.Implementation of these workshops may aid in educat-ing PCPs about best practice in the provision of care foryouth with behavioral and MH disorders.

REFERENCESAmerican Academy of Child and Adolescent Psychiatry. (2008).

Facts for families: Teen suicide. Retrieved from http://www.aacap.org/cs/root/facts_for_families/teen_suicide

American Academy of Child and Adolescent Psychiatry. (2010).Workforce factsheet: Supply of child and adolescent psychia-

trists. Retrieved from http://www.aacap.org/cs/root/legislative_action/aacap_workforce_fact_sheet

American Academy of Child and Adolescent Psychiatry and ADHDResource Center. (2010). ADHD FAQs: What are the conse-

quences of untreated ADHD?. Retrieved from http://www.aacap.org/cs/ADHD.ResourceCenter/adhd_faqs#question6

American Academy of Child and Adolescent Psychiatry Committeeon Health Care Access and Economics Task Force on MentalHealth. (2009). Improving mental services in primary care: Re-ducing administrative and financial barriers to access and col-laboration. Pediatrics, 128(4), 1248-1251.

American Academy of Child and Adolescent Psychiatry Work Groupon Quality Issues. (2007a). Practice parameter for the assess-ment and treatment of children with anxiety disorders. Journal

Journal of Pediatric Health Care

of the American Academy of Child and Adolescent Psychiatry,

46(2), 267-283.American Academy of Child and Adolescent Psychiatry Work Group

on Quality Issues. (2007b). Practice parameter for the assess-ment and treatment of children and adolescents with attentiondeficit/hyperactivity disorder. Journal of the American Academy

of Child and Adolescent Psychiatry, 46(7), 894-921.American Academy of Child and Adolescent Psychiatry Work Group

on Quality Issues. (2007c). Practice parameter for the assess-ment and treatment of children and adolescents with depres-sive disorders. Journal of the American Academy of Child and

Adolescent Psychiatry, 46(11), 1503-1526.American Academy of Child and Adolescent Psychiatry Work Group

on Quality Issues. (2009). Practice parameter on the use of psy-chotropic medication in children and adolescents. Journal ofthe American Academy of Child and Adolescent Psychiatry,

48(9), 961-973.American Academy of Pediatrics. (2002). The medical home. Pediat-

rics, 110(1), 184-186.Barclay, L. (2009). Financial barriers to children�s mental health ser-

vices should be removed. Retrieved from http://www.medscape.com/viewarticle/590648

Biederman, J., & Farone, S. (2006). The effects of attention deficit/

hyperactivity disorder onemployment andhousehold income. Re-trieved from www.ncbi.nlm.nih.gov/pmc/articles/PMC1781280

Bridge, J. A., Iyengar, S., Salary, C. B., Barbe, R. P., Birmaher, B.,Pincus, H. A., . Brent, D. A. (2007). Clinical response andrisk for reported suicidal ideation and suicide attempts in pedi-atric antidepressant treatment: A meta-analysis of randomizedcontrolled trials. Journal of the American Medical Association,

297(15), 1683-1696.Bromley, P. (2010). Online learning: Anywhere anytime education for

specialist nursing. Neonatal, Paediatric and Child Health Nurs-

ing, 13(3), 2-6.Campo, J. (2009). Youth suicide prevention: Does access to care

matter? Current Opinion in Pediatrics, 21, 628-634.Carbone, P., Behl, D., Azor, V., & Murphy, N. (2010). The medical

home for children with autism spectrum disorders: Parent andpediatrician perspectives. Journal of Autism andDevelopmental

Disorders, 40, 317-324.Carbone, P., Farley, M., & Davis, T. (2010). Primary care for children

with autism. American Family Physician, 81(4), 453-460.Centers for Disease Control and Prevention, Autism and Develop-

mental Disabilities Monitoring Network Surveillance Year 2008Principal Investigators. (2012). Prevalence of autism spectrumdisorders—autism and developmental disabilities monitoringnetwork, 14 sites, United States, 2008.Morbidity and Mortality

Weekly Report Surveillance Summary, 61(SS03), 1-19.Chikotas, N. E. (2008). Supporting the use of problem-based learning

in the education of the nurse practitioner. Nursing Education

Perspectives, 29(6), 359-362.Church, E. A., Heath, O. J., Curran, V. R., Bethune, C., Callanan,

T. S., & Cornish, P. (2010). Rural professionals� perceptions ofinterprofessional continuing education in mental health. Healthand Social Care in the Community, 18(4), 433-443.

Connor, D., McLaughlin, T., Jeffers-Terry, M., O�Brien, W., Stille, C.,Young, L., & Antonelli, R. (2006). Targeted child psychiatric ser-vices: A new model of pediatric primary clinician-child psychia-try collaborative care. Clinical Pediatrics, 45, 423-434.

Foley, M. (2011). A comparison of family adversity and family dys-function in families of children with attention deficit hyperactivitydisorder (ADHD) and families of children without ADHD. Journalfor Specialists in Pediatric Nursing, 16, 39-49.

Foy, J., Kelleher, K., & Laraque, D. (2010). Enhancing pediatric men-tal health care: Strategies for preparing a primary care practice.Pediatrics, 125(Suppl. 3), S87-S108.

Golnik,A., Ireland,M.,&Borowsky, I.W. (2009).Medical homes forchil-drenwithautism:Aphysiciansurvey.Pediatrics, 123(3), 966-971.

www.jpedhc.org

Golnik, A., & Maccabee-Ryaboy, N. (2010). Autism: Clinical pearls forprimary care. Contemporary Pediatrics, 27(11), 42-60.

Hassanin, H., Harbi, A., Saif, A., Davis, J., Easa, D., & Harrigan, R.(2010). Changes in antidepressant medications prescribingtrends in children and adolescents in Hawai�i following theFDA black box warning. Hawai�i Medical Journal, 69, 17-19.

Heneghan, A., Garner, A. S., Storfer-Isser, A., Kortepeter, K., Stein,R. E., & Horwitz, S. M. (2008). Pediatricians� role in providingmental health care for children and adolescents: Do pediatri-cians and child and adolescent psychiatrists agree? Journal

of Developmental and Behavioral Pediatrics, 29(4), 262-269.Horsfall, J., & Cleary, M. (2008). Planning and facilitating workshops.

(2008). Journal of Continuing Education in Nursing, 39(11),511-516.

Kelleher, K., Campo, J., & Gardner, W. (2006). Management of pedi-atric mental disorders in primary care: Where are we nowand where are we going? Current Opinion in Pediatrics, 18,649-653.

Kerfoot, B. P., Fu, Y., Baker, H., Connelly, D., Ritchey, M. L., &Genega, E. M. (2010). Online spaced education generatestransfer and improves long-term retention of diagnostic skills:A randomized controlled trial. Journal of the American Collec-

tion of Surgery, 211, 331-337.Kolko, D., Campo, J., Kelleher, K., & Cheng, Y. (2010). Improving

access to care and clinical outcome for pediatric behavioralproblems: A randomized trial of a nurse-administered interven-tion in primary care. Journal of Developmental and Behavioral

Pediatrics, 31(5), 393-404.Lovrin, M. (2009). Treatment of major depression in adolescents:

Weighing the evidence of risk and benefit in light of black boxwarnings. Journal of Child and Adolescent Psychiatric Nursing,

22(2), 63-68.Matzie, K. A., Kerfoot, B. P., Hafler, J. P., & Breen, E. M. (2009).

Spaced education improves the feedback that surgical resi-dents give to medical students: A randomized trial. American

Journal of Surgery, 197(2), 252-257.Melnyk, B., & Moldenhauer, Z. (2006). The KySS guide to child and

adolescent mental health screening, early intervention and

health promotion. Cherry Hill, NJ: National Association of Pedi-atric Nurse Practitioners.

Merikangas, K. R., He, J., Burstein, M., Swanson, S. A., Avenevoli,S., Cui, L., . Swendsen, J. (2010). Lifetime prevalence ofmental disorders in US adolescents: Results from the nationalcomorbidity study-adolescent supplement (NCS-A). Journal

of the American Academy of Child and Adolescent Psychiatry,

49(10), 980-989.Merikangas, K. R., He, J., Burstein, M., Swendsen, J., Avenevoli, S.,

Case, B., . Olfson, M. (2011). Service utilization for lifetimemental disorders in U.S. adolescents: Results of the nationalcomorbidity survey—adolescent supplement (NCS-A). Journalof the Academy of Child and Adolescent Psychiatry, 50(1),32-45.

National Alliance on Mental Illness. (2011). Integrating mental health

and pediatric primary care. Retrieved from http://www.nami.org/Content/ContentGroups/CAAC/FG-Integrating.pdf

National Association of Pediatric Nurse Practitioners. (2007).NAPNAP position statement on integration of mental healthcare in pediatric primary care settings. Journal of Pediatric

Health Care, 21, 29A-30A.National Institute for Health Care Management. (2009). Strategies to

support the integration of mental health into pediatric primary

care (Issue paper). Washington, DC: National Institute forHealth Care Management.

Price, B. (2010). Disseminating best practice through workshops.Nursing Standard, 24(28), 35-41.

Schlesinger, A. (2008). Pediatric primary care providers and depres-sion in community settings. Journal of the American Academy

of Child and Adolescent Psychiatry, 47(9), 976.

January/February 2014 33

Shaw, T., Long, A., Chopra, S., & Price Kerfoot, B. (2011). Impact onclinical behavior of face-to-face continuing medical educationblendedwith online spaced education: A randomized controlledtrial. Journal of Continuing Education in the Health Professions,

31(2), 103-108.Smith, M. J. (2008). Public psychiatry: A neglected professional role?

Advances in Psychiatric Treatment, 14, 339-346.Subcommittee on Attention-Deficit/Hyperactivity Disorder, Steering

Committee on Quality Improvement Management. (2011).ADHD clinical practice guideline for the diagnosis, evaluation,

and treatment of attention-deficit/hyperactivity disorder in

children and adolescents. Retrieved from http://pediatrics.aappublications.org/content/early/2011/10/14/peds.2011-2654

TeenScreen National Center for Mental Health Checkups at Colum-bia University. (2003). For primary care providers. Retrievedfrom http://www.teenscreen.org/resources/providers/

The REACH Institute. (2010). Core training programs. Retrieved fromhttp://www.thereachinstitute.org/

34 Volume 28 � Number 1

Thomas,K.C., Ellis,A.R.,Konrad, T.R.,Holzer,C.E., &Morissey, J.P.(2009). County-level estimates of mental health professional

shortage in the United States. Retrieved from http://ps.psychiatryonline.org/article/aspx?articleID=100819

Thomas, C. R., & Holzer, C. E. (2006). The continuing shortageof child and adolescent psychiatrists. Journal of the

American Academy of Child and Adolescent Psychiatry, 45(9),1023-1031.

United States Department of Health and Human Services. (2010).Mental health and mental disorders. Retrieved from www.healthypeople.gov/2020/topicsobjectives2020/overview.aspx?topicid=28

United States Department of Health and Human Services Health Re-sources and Services Administration. (2012). Find shortage

areas: MUA/P by state and county. Retrieved from http://muafind.hrsa.gov/index.aspx

United States Preventive Services Task Force. (2009).Major depres-

sive disorder in children and adolescents. Retrieved from http://www.uspreventiveservicestaskforce.org/uspstf/uspschdepr.htm

Journal of Pediatric Health Care