Greater Columbia Behavioral Health - GCBH · Summary results of Greater Columbia Behavioral Health PIP validation ... Validation scores by standard for clinical PIP, Lowered PRISM

Greater Columbia Behavioral Health

Division of Behavioral Health and Recovery External Quality Review Report

November 2013

Contract #0835-33311 Presented by

Acumentra Health 2020 SW Fourth Avenue, Suite 520 Portland, Oregon 97201-4960 Phone 503-279-0100 Fax 503-279-0190 DBHR-EQR-WA-13-9

Greater Columbia Behavioral Health

2013 External Quality Review Report

Performance Improvement Project Validation

Information Systems Capabilities Assessment

Encounter Data Validation

Clinical Record Review

November 2013

Presented to the Division of Behavioral Health and Recovery

Acumentra Health prepared this report under contract with the Division of Behavioral Health

and Recovery (Contract No. 0835-33311).

Director, State and Private Services .........Michael Cooper, RN, MN

Account Manager .....................................Jody Carson, RN, MSW, CPHQ

Project Manager–Monitoring ...................Laureen Oskochil, MPH, CHC, CPHQ

Project Manager–Validation ....................Amy Pfleiger, CISA

Project Coordinator ..................................Ricci Rimpau, RN, CHC, CPHQ

Mental Health QI Specialist .....................Nancy Siegel, PA-C, MPH

Mental Health QI Specialist .....................Erin Schwartz, PhD, LPC

Research Analyst .....................................Sara Hallvik, MPH

Information Systems Analyst ...................Christine Ryan

Writer/Editor ............................................Greg Martin

Production Assistant ................................Ellen Gehringer

2013 Greater Columbia Behavioral Health EQR Report: Table of Contents

i Acumentra Health November 2013

TABLE OF CONTENTS

Executive Summary .....................................................................................................................................1

Introduction ..................................................................................................................................................5

EQR activities ........................................................................................................................................5

Performance Improvement Project (PIP) Validation ...................................................................................6

PIP review procedures ...........................................................................................................................6

Review results for nonclinical PIP: Lowered Inpatient Readmission Rates in a High Risk

Population through the Development of Enhanced Communication with Inpatient Providers .............8

Review results for clinical PIP: Lowered PRISM Scores in a High Medical Risk Psychiatric

Inpatient Population Through the Integration of Health Information into Mental Health Records ....14

Compliance Review Follow-Up ................................................................................................................21

Information Systems Capabilities Assessment ..........................................................................................23

Summary of review results ..................................................................................................................24

ISCA Section 1: Data Processing Procedures and Personnel ..............................................................25

ISCA Section 2: Data Acquisition Capabilities ...................................................................................29

Encounter Data Validation .........................................................................................................................32

Greater Columbia Behavioral Health’s EDV procedure .....................................................................32

Validation results .................................................................................................................................34

Discussion and recommendations ........................................................................................................38

Clinical Record Review .............................................................................................................................39

Review results ......................................................................................................................................39

Discussion and recommendations ........................................................................................................39

Appendix A. PIP Validation and Scoring Protocol ................................................................................ A-1

Appendix B. ISCA Methodology ............................................................................................................B-1

Appendix C. Encounter Data Validation Procedures ..............................................................................C-1

2013 Greater Columbia Behavioral Health EQR Report: Index of Tables and Figures

ii Acumentra Health November 2013

INDEX OF TABLES AND FIGURES

Table 1. Standards for PIP validation ......................................................................................................7

Table 2. PIP scoring ranges......................................................................................................................7

Table 3. Results of compliance review follow-up for Greater Columbia Behavioral Health ................22

Table 4. Scoring scheme for ISCA standards ........................................................................................23

Table 5. Weighted average scores and ratings on ISCA sections ..........................................................24

Table 6. Results of 2013 electronic data checks ....................................................................................35

Table 7. Results of encounter data validation for Greater Columbia Behavioral Health ......................36

Table 8. Results of Golden Thread analysis for Greater Columbia Behavioral Health .........................37

Table 9. Results of clinical record review for Greater Columbia Behavioral Health ............................41

Figure 1. Summary results of Greater Columbia Behavioral Health PIP validation ................................2

Figure 2. Summary results of Greater Columbia Behavioral Health ISCA review ..................................3

Figure 3. Validation scores by standard for nonclinical PIP, Lowered Inpatient Readmission Rates

in a High Risk Population through the Development of Enhanced Communication with

Inpatient Providers .....................................................................................................................8

Figure 4. Validation scores by standard for clinical PIP, Lowered PRISM Scores in a High Medical

Risk Psychiatric Inpatient Population Through the Integration of Health Information into

Mental Health Records ............................................................................................................14

2013 Greater Columbia Behavioral Health EQR Report: Executive Summary

1 Acumentra Health November 2013

EXECUTIVE SUMMARY

The Division of Behavioral Health and Recovery

(DBHR) contracts with Acumentra Health to

perform an annual external quality review (EQR)

of managed mental health services for Medicaid

enrollees in Washington, in fulfillment of federal

requirements under 42 CFR §438.350.

This report summarizes the 2013 review of

Greater Columbia Behavioral Health (GCBH),

one of 11 regional support networks (RSNs) with

which DBHR contracts to deliver managed mental

health services. This year’s review includes:

evaluation of the RSN’s performance

improvement projects (PIPs)

follow-up review of the RSN’s compliance

with federal and state regulations and

contract provisions governing managed

care operations

an Information Systems Capabilities

Assessment (ISCA)

encounter data validation and a review of

clinical records

The reviews rate GCBH’s overall performance in

2012, identify strengths and opportunities for

improvement, and offer specific recommendations

to address deficiencies. The results summarized

below are presented in more detail in the main

body of the report.

PIP evaluation results

Because RSNs begin their PIPs at different times,

and because PIPs are typically multi-year projects,

the studies may be in different stages at the time

of the EQR evaluation. As ongoing projects, the

PIPs may not meet all standards the first year, but

a PIP is expected to achieve better scores as the

project progresses, eventually reaching full

compliance.

Acumentra Health reviewed one nonclinical and

one clinical PIP conducted by GCBH:

1. Nonclinical—Lowered Inpatient

Readmission Rates in a High Risk

Population through the Development of

Enhanced Communication with

Inpatient Providers: Feedback from local

stakeholders revealed concerns about

inpatient recidivism among children

enrolled with GCBH. Between October

2011, when GCBH established its in-house

Authorization Center, and December 2012,

GCBH calculated a 90-day readmission

rate of 28% for children. For this first-year

PIP, GCBH intends to implement the use of

the Child Inpatient Admission Review

questionnaire for each child authorized for

an inpatient stay. The purpose of the

questionnaire is to facilitate enhanced

communication between GCBH’s in-house

Authorization Center staff and inpatient

providers’ utilization personnel, with the

ultimate goal of reducing the readmission

rate for children.

2. Clinical—Lowered PRISM Scores in a

High Medical Risk Psychiatric

Inpatient Population Through the

Integration of Health Information into

Mental Health Records: This first-year

PIP focuses on integrating physical health

information into mental health treatment

plans as a means of addressing the under-

treatment of physical health needs. The

RSN provided no evidence as to the nature

and extent of this problem at the local

Medicaid level, and thus could not discuss

how its intervention strategy addresses

identified barriers or is expected to

improve the study indicator (risk scores

for mental health enrollees). It is also not

clear whether the selected indicator is the

most appropriate measure of integration of

physical health information into mental

health treatment plans.



53

43

0

30

60

90

Nonclinical Clinical

Sco

re

Figure 1. Summary results of Greater Columbia Behavioral Health PIP validation.

As shown in Figure 1, the nonclinical PIP scored

53 on a scale of 90, and the clinical PIP scored 43

out of 90, both earning a Partially Met rating.

GCBH presented little data to support the design

and implementation of the nonclinical PIP. The

RSN needs to support the selection of the study

topic and present more information regarding the

codes used to pull administrative data, procedures

for data validation, why GCBH chose the specific

intervention, and a plan for tracking and

monitoring the intervention.

With regard to the clinical PIP, GCBH did not

fully document its reasons for selecting the study

topic, indicator, and intervention. In its next

iteration, GCBH needs to focus on key PIP

concepts—relevance of the topic to the local

Medicaid population, validity of the study

indicator, and barrier analysis to support the

selection of the intervention.

Compliance follow-up results

In 2013, Acumentra Health reviewed GCBH’s

response to the specific findings of the 2012

compliance review for which DBHR required the

RSN to perform corrective action. The 2012

review addressed compliance with federal and

state standards in eight separate areas.

The follow-up review found that GCBH had

addressed all four findings from the 2012 review.

Notably, the RSN completed a new quality

management plan, rewrote several policies,

formalized its credentialing/recredentialing

process, and began using computerized auditing

tools to establish baselines and monitor for over-

and underutilization of services.

ISCA results

Acumentra Health examined GCBH’s

information systems and data processing and

reporting procedures to determine the extent to

which they supported the production of valid and

reliable state performance measures and the

capacity to manage care of RSN enrollees.

The ISCA found that GCBH partially met the

federal standards related to data processing

procedures and personnel, and fully met the

standards related to data acquisition capabilities.

The review revealed deficiencies with regard to

data security and eligibility verification.

Figure 2 depicts GCBH’s scores on individual

ISCA subsections. For additional detail,

including strengths and recommendations for

improvement, see the report section beginning

on page 23.

Substantially met

Fully met

Partially met

Not met

Minimally met



2.32.2

2.6

1.5

2.9 2.7

2.22.3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Se

cti

on

sc

ore

Figure 2. Summary results of Greater Columbia Behavioral Health ISCA review.

Encounter data validation results

DBHR requires each RSN to conduct an annual

encounter data validation (EDV) to determine the

accuracy of encounter data submitted by network

providers. As an independent check of the RSNs’

results, Acumentra Health audited and verified

the EDV process for each RSN in 2013.

Electronic checks of the data elements making up

GCBH’s total outpatient and demographic data

generally found complete data in mandatory

fields, but 13.2% of records contained out-of-

range ethnicity values, and a small percentage of

records had a duplicate claim ID.

Acumentra Health’s review of 394 outpatient

encounter records revealed that 79.6% had a

service code that matched the service described in

the chart note. Provider type, service date and

location, and procedure code matched in 80% to

93% of encounters. Considering demographic

data, the chart information matched the state data

in 99.1% of records for first and last name, and

matched in all records for date of birth. Matching

rates were lower for ethnicity and language.

Golden Thread analysis

In conjunction with the EDV, Acumentra Health

reviewed clinical records to determine providers’

adherence to the “Golden Thread” of clinical

documentation, tying together the mental health

diagnosis, treatment plan, and progress notes.

Reviewers examined whether the assessment in

the enrollee’s chart substantiated the diagnosis,

whether the treatment plan was consistent with the

diagnosis, and whether progress notes addressed

goals and interventions in the treatment plan.

Assessments. Most mental health assessments

were well written for both children and adults.

They included documentation of the presenting

problem, medical necessity for treatment, clinical

formulation for diagnosis, and services.

Treatment plans. Although the majority of the

children’s treatment plans included interventions

and goals consistent with issues identified in the

assessment, only 67% of the adult treatment plans

included interventions, services, and measurable

goals that were consistent with the assessment.

Not met

Partially met

Fully met



GCBH needs to provide guidance to the

agencies to ensure that treatment plans

specify measurable goals, expected

outcomes, service(s) indicated, and

specific interventions consistent with the

enrollee’s assessment.

Progress notes. The progress notes for children

were consistent and well written. They typically

addressed interventions identified in the treatment

plan and the child’s progress toward meeting the

stated goals. However, as with the treatment

plans, the adult progress notes showed room for

improvement, as they did not consistently address

interventions identified in the treatment plan and

progress toward meeting the stated goals.

GCBH needs to provide direction to the

agencies to ensure that progress notes

clearly document the goals, treatment

interventions, enrollee’s response to the

interventions, and progress toward

meeting the goals.

Clinical record review results

Also in conjunction with the EDV, Acumentra

Health reviewed clinical records at four outpatient

provider agencies to assess compliance with the

mental healthcare criteria defined by DBHR. This

study focused on the degree to which the RSN’s

system of care adhered to the principles of the

Children’s Mental Health Redesign, including the

principles defined in the interim settlement of T.R.

v. Dreyfus, regarding uniform screening and

assessment of children with serious emotional

disturbances.

Assessments: Clinicians at GCBH’s provider

agencies did a good job of documenting the

child’s home environment and systems of support.

The percentage of applicable assessments that

addressed developmental and sensory impairment,

cultural and language issues, and justifying the

diagnosis all fell within the acceptable range.

However, nearly one-third of the assessments were

more than one year old.

Acumentra Health recommends updating

enrollee assessments at least annually to

document changes in the enrollee’s

functioning and life circumstances.

Treatment plans: Most treatment plans reflected

information included in the assessments. The

majority of treatment goals were based on the

children’s strengths. Most records documented

involvement of family members and support

systems in treatment. However, only 64% of the

treatment plans incorporated coordination with

other agencies into the treatment objectives.

GCBH should ensure that agencies

work with their clinicians to coordinate

care with other agencies involved in the

child’s life.

Nearly two-thirds of the treatment plans did not

include a multi-disciplinary team-based approach

to treatment.

GCBH needs to offer guidance to the

agencies to ensure that children’s

treatment includes a team-based

approach when necessary.

Progress notes: All progress notes demonstrated

that the child received unconditional treatment.

The majority of progress notes documented use of

strength-based interventions and activities as

identified in the treatment plans. Team-based

services and coordination with other agencies

were missing from many progress notes. Only

55% of progress notes indicated that services were

team-based and that care was coordinated with

other agencies and systems.

GCBH needs to ensure that services for

children are team-based and that care

coordination occurs. These services

must be documented in the progress

notes to reflect a multi-disciplinary

team-based approach.

2013 Greater Columbia Behavioral Health EQR Report: Introduction


INTRODUCTION

This report summarizes the results of the 2013

review of GCBH, a mental health RSN that serves

Medicaid recipients. Acumentra Health performed

the review in its capacity as DBHR’s External

Quality Review Organization (EQRO).

Currently, DBHR contracts with 11 RSNs to

deliver mental health services for Medicaid

enrollees through managed care. The RSNs, in

turn, contract with provider groups, including

community mental health programs and private

nonprofit agencies and hospitals, to deliver

treatment services. The RSNs must ensure that

services are delivered in a manner that complies

with legal, contractual, and regulatory standards

for effective care.

GCBH, headquartered in Kennewick, is a

government consortium providing public mental

health services for 10 counties and the Yakama

Nation in south central Washington. A citizen’s

advisory board advises the GCBH board of

directors, reviews and provides comments and/or

recommendations on plans and policies, and

serves on RSN workgroups and committees.

During 2012, GCBH had about 190,000 enrollees

in its service area.

EQR activities

42 CFR §438.358 specifies three mandatory

activities that the EQR must address in a manner

consistent with protocols established by the

Centers for Medicare & Medicaid Services

(CMS):

a review every three years of health plan

compliance with federal and state

regulations and contract provisions

regarding access to care, structure and

operation, and quality measurement and

improvement

annual validation of PIPs, a required

element of health plans’ quality

improvement (QI) programs

annual validation of performance

measures reported by plans or calculated

by the state, including an ISCA

Acumentra Health conducted the compliance

review for each RSN during 2011–2012, and

conducted the ISCA for each RSN and for

DBHR in 2013. In addition, Acumentra Health

conducted the PIP validation, an encounter data

validation, and a clinical record review for each

RSN in 2013. Together, these activities

addressed the following questions:

1. Does the RSN meet CMS regulatory

requirements?

2. Does the RSN meet the requirements of its

contract with the state?

3. Does the RSN monitor and oversee

contracted providers in their performance

of any delegated activities to ensure

regulatory and contractual compliance?

4. Does the RSN conduct the two required

PIPs, and are they valid?

5. Does the RSN’s information technology

infrastructure support the production and

reporting of valid and reliable performance

measures?

Review procedures for each activity were adapted

from the applicable CMS protocol:

EQR Protocol 1: Assessment of

Compliance with Medicaid Managed Care

Regulations. Version 2.0, September 2012

EQR Protocol 3: Validating Performance

Improvement Projects (PIPs). Version 2.0,

September 2012

Appendix V: Information Systems

Capabilities Assessment. September 2012

The scoring plan for each activity was adapted

from CMS guidelines, using a DBHR-approved

weighting system developed by Acumentra

Health.

2013 Greater Columbia Behavioral Health EQR Report: PIP Overview


PERFORMANCE IMPROVEMENT

PROJECT VALIDATION

Under 42 CFR §438.240(d), a managed care

organization that serves Medicaid enrollees must

have an ongoing program of PIPs that focus on

improving clinical care and nonclinical aspects of

service delivery. The PIPs enable the organization

to assess and improve the processes and, in turn,

the outcomes of care.

PIPs are validated each year as part of the EQR to

ensure that the projects are designed, conducted,

and reported according to accepted methods,

establishing confidence in the reported results.

The PIPs must include:

measurement of performance using

objective quality indicators

implementation of system interventions to

improve quality

evaluation and initiation of the

interventions

planning and initiation of activities for

increasing or sustaining improvement

Through repeated measurement of the selected

quality indicators, a PIP is expected to

demonstrate meaningful change in performance

relative to the performance observed during

baseline measurement.

Acumentra Health has validated the Washington

RSNs’ PIPs each year since 2008. Most RSNs

have carried their individual PIP topics forward

for at least several years, enabling Acumentra

Health to evaluate their progress toward achieving

sustained improvement.

Because RSNs begin their PIPs at different times,

the studies may be in different stages at the time

of the EQR evaluation. Some may be underway

but not yet complete; others may have progressed

to collecting baseline and remeasurement data;

still others may have progressed to multiple

remeasurements. The stage of the PIP at review

determines the level of analysis that Acumentra

Health applies.

PIP review procedures

Through document review and onsite interviews,

Acumentra Health reviews PIPs for these elements:

a written project plan with a study design,

an analysis plan, and a summary of results

a clear, concise statement of the topic being

studied, the specific questions the study is

designed to address, and the quantifiable

indicators that will answer those questions

a clear statement of the improvement

strategies, their impact on the study

question, and how that impact will be

assessed and measured

evidence that the intervention services and

materials are culturally and linguistically

appropriate, per the 2012 CMS protocol

an analysis plan that addresses project

objectives, defines indicators clearly,

specifies the population being studied,

identifies data sources and/or the data

collection procedure, and discusses the

methods for analyzing the data and

performing statistical tests

if applicable, a sampling methodology that

yields a representative sample

if the data collection involves clinical chart

review, a check on inter-rater reliability

use of validation procedures at the point of

data entry and within the database to verify

that data used for population-based analysis

are complete and accurate

a summary of results covering all data

collection and analysis, explaining

limitations in the data and methodologies

and discussing whether the intervention(s)

resulted in improvements

PIP scoring

The PIP scoring methodology, adapted from the

CMS protocol for this activity and approved by

DBHR, involves rating the RSN’s performance

on as many as 10 standards, listed in Table 1.

Appendix A defines in detail the specific criteria

used to evaluate performance.

2013 Greater Columbia Behavioral Health EQR Report: PIP Overview


Each standard has a potential score of 100 points

for full compliance. The total points earned for

each standard are weighted and combined to

determine an overall PIP score. The overall score

is weighted 90% for demonstrable improvement in

the first year (Standards 1–8) and 10% for

sustained improvement in later years (Standards

9–10). Thus, for a PIP that has completed one

remeasurement, the maximum overall project score

is 90 points. (Note: In years before 2012, the

maximum score for these PIPs was 80 points.) If

the PIP has progressed to at least a second

remeasurement, enabling reviewers to assess

sustained improvement, the maximum overall

project score is 100 points.

Table 1. Standards for PIP validation.

Demonstrable improvement

1 Selected study topic is relevant and prioritized

2 Study question is clearly defined

3 Study indicator is objective and measurable

4 Study population is clearly defined and, if a sample is used, appropriate methodology is used

5 Data collection process ensures valid and reliable data

6 Improvement strategy is designed to change performance based on the quality indicator

7 Data are analyzed and results interpreted according to generally accepted methods

8 Reported improvement represents “real” change

Sustained improvement

9 The RSN has documented additional or ongoing interventions or modifications

10 The RSN has sustained the documented improvement

Table 2 shows the compliance ratings and

associated scoring ranges for PIPs graded on the

90-point and 100-point scales. Appendix A

presents a sample scoring worksheet. Note: these

compliance rating ranges for the overall PIP score

are different from the ranges used in assessing

compliance for individual PIP standards; see

Appendix A.

Per the approved protocol, Acumentra Health

scores all PIPs according to the same criteria,

regardless of the stage of completion. As ongoing

multi-year QI projects, the PIPs may not meet all

standards the first year, but a PIP is expected to

achieve better scores as project activities progress,

eventually reaching full compliance.

Table 2. PIP scoring ranges.

Compliance rating Description

100-point scale

90-point scale

Fully met Meets or exceeds all requirements 80–100 72–90

Substantially met Meets essential requirements, has minor deficiencies 60–79 54–71

Partially met Meets essential requirements in most, but not all, areas 40–59 36–53

Minimally met Marginally meets requirements 20–39 18–35

Not met Does not meet essential requirements 0–19 0–17

2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation


90

100

95

60 60

35

0 00

20

40

60

80

100

1 2 3 4 5 6 7 8

Score

Standard

Review results for nonclinical PIP: Lowered Inpatient Readmission Rates in a High Risk Population through the Development of Enhanced Communication with Inpatient Providers

In response to the statewide mandate for RSNs to

provide services and treatment to children within

their communities, GCBH sought feedback from

local stakeholders about potential areas for

improvement related to children’s mental health.

The stakeholders expressed specific concerns

about inpatient recidivism. Between October

2011, when GCBH established its in-house

Authorization Center, and December 2012,

GCBH calculated a 90-day readmission rate of

28% for children. GCBH cited research studies

showing that effective discharge planning and the

establishment of better collaborative relationships

between managed care and inpatient providers are

critical in reducing hospital readmissions.

For this first-year PIP, GCBH intends to implement

the use of the Child Inpatient Admission Review

questionnaire for each child authorized for an

inpatient stay. The purpose of the questionnaire is

to facilitate enhanced communication between

GCBH’s in-house Authorization Center staff and

inpatient providers’ utilization personnel, with the

ultimate goal of reducing the readmission rate for

children.

GCBH presented little data to support the design

and implementation of this PIP. The RSN needs to

present data to support the selection of the study

topic, and more information regarding the codes

used to pull administrative data, procedures for

data validation, why GCBH chose the specific

intervention, and a plan for tracking and

monitoring the intervention.

The overall weighted score for this PIP is 53,

based on a scale of 90, resulting in a compliance

rating of Partially Met.

Figure 3 shows the score for each of the eight

validation standards reviewed for this project.

Figure 3. Validation scores by standard for nonclinical PIP, Lowered Inpatient Readmission Rates in a High Risk Population through the Development of Enhanced Communication with Inpatient Providers.

Substantially met

Fully met

Partially met

Not met

Minimally met



Standard 1: Study Topic Score: 90 (Substantially met)

To meet Standard 1, the RSN needs to establish

the importance of the study topic in general and

present local data to demonstrate that the topic

applies to a large or high-risk portion of the

Medicaid population; and demonstrate that a

systematic selection and prioritization process

was used in choosing the topic.

GCBH cited several reasons for selecting this

topic, including: (a) a contractual requirement for

each RSN to conduct at least one PIP focused on

children, with the goal of providing services and

treatment within community settings; (b) concerns

expressed by stakeholders with regard to inpatient

readmissions for children; and (c) 112 inpatient

readmissions within 90 days of discharge out of

401 admissions that occurred between October

2011 and December 2012, resulting in a 28%

readmission rate for that period. GCBH did not

break down these readmissions among acute care

settings, the state hospital, and Children’s Long-

Term Inpatient Program (CLIP) facilities, nor did

the RSN present other data that would enable a

comparison with the local readmission rate.

GCBH cited studies showing that “effective

discharge planning is critical to preventing

unnecessary inpatient readmissions” and that

“improved relations between managed care and

inpatient providers support better discharge

outcomes.”

Before opening its in-house Authorization Center

in October 2011, GCBH lacked access to the data

necessary to determine whether the RSN was

adhering to its philosophy that “inpatient services

should not be used when there are viable

alternatives that address a consumer’s needs.” The

RSN and affiliated stakeholders elected to pursue

this study topic based on concerns about hospital

readmissions for children, improved access to data

related to hospitalizations, and the ability of

GCBH’s Authorization Center staff to engage

directly with inpatient providers.

GCBH stated that readmission rates indicate the

“quality or adequacy of treatment services that the

individual received during the previous

hospitalization.” For this project, GCBH plans to

implement the use of a questionnaire intended to

facilitate enhanced communication between the

Authorization Center staff and inpatient providers.

GCBH reported that “studies suggest that benefits

to this approach include higher individual client

outcomes, identification of types of common

debilitating gaps in services, identification of

trends, and the opportunity to increase system-

wide awareness.”

To fully meet this standard, GCBH needs to

provide additional data about the use of inpatient

facilities included in the RSN’s readmission rate,

and demonstrate that hospital readmissions for

children is a significant area of need.

Standard 2: Study Question Score: 100 (Fully met)

To meet Standard 2, the RSN needs to present a

study question that provides a clear framework

for data collection, analysis, and interpretation.

The study question should refer to the proposed

intervention, a study population (denominator), a

measure (numerator), a metric (e.g., average,

percentage), and a direction of desired change.

GCBH presented this study question:

“Does enhanced communication with inpatient

providers, via the implementation of a child

inpatient admission questionnaire by GCBH

Authorization Center staff at the time of

authorization/admission, decrease the proportion

of children readmitted to community inpatient

services within 90 days of his/her inpatient

discharge?”

The question establishes a clear framework for

analysis, including an intervention (enhanced

communication), a study population (children

discharged from community inpatient services), a

measure (readmitted within 90 days), a metric

(proportion), and a direction of change (decrease).

GCBH fully meets this standard.



Standard 3: Study Indicator Score: 95 (Substantially met)

To meet Standard 3, the RSN needs to define the

measure (numerator) and study population

(denominator); define key terms; and discuss the

basis for adopting the indicator as a valid proxy

for enrollee outcomes, satisfaction, or quality of

care.

GCBH defined the elements of the study indicator

as follows.

Denominator: The number of children in a given

measurement period who were discharged from

inpatient care in a community setting, and who

were Medicaid-eligible at the time the inpatient

episode of care commenced.

Numerator: The number of children in the

denominator who were readmitted to a

community inpatient setting between the 2nd

and

90th

calendar day following discharge from a prior

episode of inpatient care.

GCBH defined the following key terms.

Authorized: Includes only children approved by

a staff member of the GCBH Authorization

Center to receive inpatient services.

Children: Individuals 20 years or younger at the

time of admission.

Episode of inpatient care: A span of days during

which the enrollee received inpatient services in a

single inpatient setting or in multiple settings

where admission to a new setting occurred within

1 calendar day of discharge from the previous

one. Defining readmissions occurring within 1

calendar day as “transfers” is a common practice

when calculating readmission rates.

Community inpatient setting: Any inpatient

entity, including evaluation and treatment (E&T)

facilities, that provides inpatient mental health

services. Includes out-of-RSN and out-of-state

providers and excludes settings to which

admission is not authorized by the GCBH

Authorization Center staff (e.g., state hospitals,

CLIP facilities).

Discharged: The end of an episode of inpatient

care.

Medicaid eligibility: Eligibility status determined

by GCBH Authorization Center staff at the time

of admission on the basis of information available

to them via Provider One.

Readmission: A new episode of inpatient care

that occurs when a patient is readmitted to a

psychiatric inpatient facility within 90 days of a

previous discharge.

GCBH noted that its definition of Medicaid

eligibility could lead to excluding children from

the study indicator who are eligible for Medicaid

at the time of admission, but not at the time of

readmission. However, GCBH stated that this

would be a rare occurrence.

In defining readmission, GCBH did not indicate

that the admission needed to occur between the

2nd

and 90th

day following a previous discharge to

be included in the study indicator.

GCBH reported that it selected the study

indicator, readmission to an inpatient setting

within 90 days of discharge, because “numerous

studies suggest that largest risk for readmission is

within the first 30 days of discharge and remains

high for 3 months.”

GCBH stated its belief that readmission rates

indicate not only the quality of medical treatment

received during an inpatient stay, but also the

quality of discharge planning. The RSN believes

that the use of the Child Inpatient Admission

Review questionnaire by the Authorization Center

will enhance communication between the RSN

and inpatient providers. GCBH anticipates that

enhanced communication will help identify

“needs, gaps, and risk factors for readmissions,”

which they believe will lead to improved

discharge planning for consumers and, ultimately,

fewer readmissions.

To fully meet this standard, GCBH should

provide a definition of readmission consistent

with the description of the study numerator.



Standard 4: Denominator (Study Population) Data Collection Score: 60 (Partially met)

To meet Standard 4, the RSN needs to list all

inclusion and exclusion criteria for the study

population; document all data sources, including

fields, codes, and calculations; and describe data

validation procedures. If a sample is selected, the

RSN needs to describe the sampling methods.

GCBH identified inclusion criteria for the study

population, specifying data sources, tables, fields,

and calculations. The data elements include

Medicaid eligibility, age, inpatient provider,

inpatient admission, inpatient discharge, and

authorization. All data are available through

GCBH’s Consumer Information System (CIS),

which consolidates enrollee information including

Medicaid eligibility and age, which are available

from Provider One. GCBH did not report specific

codes that would indicate an inpatient admission

related to mental health.

GCBH excluded individuals from the study

population who were older than 20 years at the

time of admission to an inpatient psychiatric

setting. The study also excludes enrollees who are

admitted to CLIP or state hospital facilities,

because a different authorization process is used

for those admissions. GCBH did not discuss what

effect, if any, excluding these enrollees might

have in terms of accurately assessing the RSN’s

readmission rate for children.

GCBH provided no information related to data

validation procedures for the study population.

The report directs the reader to “see the table at

the end of the document for specifics concerning

the data elements,” but no information related to

data validation was included there.


report applicable codes for inpatient admissions

related to mental health, discuss the exclusion of

enrollees who are admitted to state hospital and

CLIP facilities, and provide information about

data validation procedures for the study

population.

Standard 5: Numerator (What Is Being Measured) Data Collection and Analysis Plan Score: 60 (Partially met)


inclusion and exclusion criteria for the numerator

(what is being measured); document all data

sources, including fields, codes, and calculations;

describe data validation procedures; and present

a clear data analysis plan, including time frames

for the measurement and intervention periods,

and an appropriate statistical test to measure

differences between the baseline and

remeasurement periods.

Readmissions to inpatient care between the 2nd

and

90th

days following discharge from a previous

inpatient admission are the only inclusion criterion

for the numerator. Admission and discharge dates

obtained through CIS are the only data collected.

GCBH did not report the mental health codes that

would indicate a psychiatric inpatient admission.

An admission date on the same day or 1 day

following a discharge date is considered a transfer

and a continuation of care. Transfers are not

included in the numerator.

GCBH did not account for the possibility that an

enrollee who had a previous admission to a

community inpatient hospital setting could be

readmitted to a different type of facility (e.g.,

CLIP) within 90 days of discharge, but would not

be counted in the readmission rate. Conversely, an

enrollee could have been discharged from a CLIP

facility or the state hospital and then been

readmitted to a community inpatient setting

within 90 days and would also not be counted in

the readmission rate.

GCBH provided no information related to data

validation procedures for the data elements

included in the numerator for this PIP.

GCBH reported the following study timeline:

Baseline: July 1, 2012–September 30, 2013

Intervention: October 1, 2013–September 30,

2014



First remeasurement: October 1, 2013–

December 31, 2014

GCBH did not provide dates for the second

remeasurement period.

To identify statistical significance, GCBH plans

to use the chi-square test for independent groups

at the p≤.05 level of probability.


report the relevant mental health codes for

inpatient admission criteria; discuss the effect of

excluding CLIP and state hospital admissions

from the readmission rate calculation; provide

information related to data validation procedures;

and provide dates for the second remeasurement

period.

Standard 6: Study Intervention Score: 35 (Minimally met)

To meet Standard 6, the RSN needs to select an

improvement strategy that will affect a wide range

of enrollees or a high-risk enrollee population,

and that is reasonably expected to result in

measurable improvement. The RSN needs to

discuss the basis for adopting the intervention;

document the implementation of the intervention,

including dates and locations of principal

activities; discuss cultural competence; and track

how effectively the intervention was implemented.

For this PIP, GCBH plans to implement the use of

a questionnaire to help facilitate communication

between Authorization Center staff and inpatient

providers, from the point of authorization through

discharge, for all children eligible for inclusion in

the study population. GCBH stated that it will use

the questionnaire to “identify a wide variety of

factors related to clinical outcomes.” Following

discharge, questionnaires will be given to the

GCBH quality manager who will track

readmissions and document trends.

Although the primary responsibility of GCBH

Authorization Center staff is to determine whether

admission or continuation of inpatient care

criteria have been met, GCBH believes that staff

are in a position to work collaboratively with

inpatient providers. This process is expected to

aid in discharge planning and, ultimately, have a

positive effect on clinical outcomes. GCBH

described Authorization Center staff as “well-

qualified clinicians who have a wealth of

knowledge about available resources,” as well as

information about children’s previous contacts

with the mental health system. GCBH believes

that the implementation of the questionnaire will

enable exploration of gaps or concerns, as well as

the availability or previous use of specific

services following discharge.

GCBH provided no information related to the

validity, reliability, or previous use of the

questionnaire to justify the expectation of

improvement based on its use. GCBH did not

report whether Authorization Center staff would

receive training related to the appropriate use of

the questionnaire. It is also not clear whether

inpatient providers, who are more directly

involved with discharge planning, are aware

that Authorization Center staff will use the

questionnaire or if they know the intent of the

questionnaire.

With regard to tracking and monitoring the

intervention, GCBH reported that the questionnaire

will have check boxes next to each item and ways

for Authorization Center staff to indicate if

categories are not applicable. The questionnaire

will also include dates of discharge and enrollee

identifiers “to allow the Quality Manager to track

and monitor data.” GCBH did not indicate whether

it intends to track completion of the questionnaire

for each child eligible for inclusion in the study

population. The RSN also did not report the

anticipated frequency of its tracking and

monitoring activities.

GCBH reported that the questionnaire includes an

item that allows for the exploration of cultural

factors that may affect treatment.


provide more information about the questionnaire,

any training required to use it, and why it use is

expected to improve readmission rates. GCBH

also needs to indicate whether inpatient providers,



who are expected to help complete the

questionnaires, are aware of the plan to use the

questionnaire and the intent behind its use.

Finally, GCBH needs to provide more

information about its plan for tracking and

monitoring the implementation of its intervention,

particularly in terms of frequency, and whether

the RSN plans to track whether the questionnaire

is completed for each eligible child.

Standard 7: Study Results Score: 0 (Not met)

To meet Standard 7, the RSN needs to present

results according to the data analysis plan,

including the study indicator, the original data

used to compute the indicator, and a statistical

test to measure differences between the baseline

and remeasurement periods; and discuss how the

intervention influenced the results.

GCBH did not report on this standard because the

study has not advanced to this stage at the time of

the PIP review.


present data for the numerator and denominator

for the baseline and remeasurement period,

present the results of statistical analyses

comparing the baseline and first remeasurement

using the appropriate test statistic and probability

level, and discuss how the intervention influenced

the study results.

Standard 8: Interpretation of Study Results Score: 0 (Not met)

To meet Standard 8, the RSN needs to assess

whether any reported improvement is “real” by

documenting that baseline and remeasurement

data were collected using the same methods and

are comparable; discuss the statistical and

clinical significance of the study results; address

barriers to improvement and lessons learned

during the PIP process; and identify confounding

factors that may have affected the results.

GCBH did not report on this standard because the

study has not advanced to this stage at the time of

the PIP review.


interpret the study results in terms of statistical

and clinical significance, evaluate the methods

used, and discuss confounding factors.

Standard 9: Study Modifications After the First Remeasurement Score: n.a.

To meet Standard 9, the RSN needs to document

modifications to the intervention, or added

interventions, planned or implemented after the

first remeasurement period; and discuss changes

in other aspects of the PIP based on lessons

learned from data analysis or barrier analysis.

Acumentra Health did not score this standard, as

the study has not progressed to a second

remeasurement.

Standard 10: Overall Study Results Score: n.a.

To meet Standard 10, the RSN needs to report

complete study results for two or more

measurement periods, including the study

indicator, original data used to compute the

indicator, and a statistical test of group

differences; and interpret the statistical and

clinical significance of the overall results, discuss

lessons learned, and determine if goals were met

and sustained improvement was achieved.


the study has not progressed to a second

remeasurement.

2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation


60

80

5560 70

45

0 00

20

40

60

80

100

1 2 3 4 5 6 7 8

Sco

re

Standard

Review results for clinical PIP: Lowered PRISM Scores in a High Medical Risk Psychiatric Inpatient Population Through the Integration of Health Information into Mental Health Records

This first-year PIP focuses on integrating physical

health information into mental health treatment

plans. GCBH cited national research findings that

under-treatment of physical health needs, as a

result of “siloed” information systems, adversely

affects mental health conditions. However, the

RSN provided no evidence as to the nature and

extent of this problem at the local Medicaid level.

Because GCBH did not analyze the gap in service,

the RSN could not discuss how its intervention

strategy addresses identified barriers or is

expected to improve the study indicator. It is also

not clear whether the selected indicator is the

most appropriate measure of integration of

physical health information into mental health

treatment plans.

During the onsite interview, GCBH discussed its

reasons for selecting the study topic, indicator,

and intervention, but the RSN did not fully

document these explanations in its report. Though

engaged and interested in learning to apply quality

improvement processes, GCBH seems hindered in

its efforts by factors such as staffing resources,

time, and provider cooperation.

Acumentra Health recognizes the challenges

facing GCBH, and encourages the RSN to focus

on key PIP concepts—relevance of the topic to

the local Medicaid population, validity of the

study indicator, and barrier analysis to support

intervention selection—in its next iteration.

The overall weighted score for this PIP is 43,

based on a scale of 90, resulting in a compliance

rating of Partially Met.

Figure 4 shows the score for each of the validation

standards reviewed for this clinical project.

Figure 4. Validation scores by standard for clinical PIP, Lowered PRISM Scores in a High Medical Risk Psychiatric Inpatient Population Through the Integration of Health Information into Mental Health Records.

Substantially met

Fully met

Partially met

Not met

Minimally met



Standard 1: Study Topic Score: 60 (Partially met)

To meet Standard 1, the RSN needs to establish

the importance of the study topic in general and

present local data to demonstrate that the topic

applies to a large or high-risk portion of the

Medicaid population; and demonstrate that a

systematic selection and prioritization process

was used in choosing the topic.

This PIP, new for 2013, focuses on the integration

of physical and behavioral health, particularly the

incorporation of physical health information into

mental health records for the purpose of care

coordination. GCBH cited several research studies

that demonstrated a link between the lack of

integration of physical health issues into mental

health treatment and increased morbidity and

mortality in the mentally ill population. The RSN

also cited anecdotal reports that mental health

providers are often uninformed as to their

patients’ medical conditions. GCBH sees this PIP

as an opportunity to “facilitate holistic treatment

of the consumer.”

On a statewide level, GCBH observed that

Washington DSHS has targeted the issue of

integration through the implementation of the

Predictive Risk Intelligence System (PRISM).

PRISM is a decision support tool “intended for

use by health care providers to identify consumers

most in need of comprehensive care coordination

based upon a risk score.”

On a local level, GCBH cited “shared voices of

concern about recidivism and its relationship to

under-addressed co-occurring medical

conditions.” GCBH discussed the role of its

Authorization Center staff in coordinating care

with inpatient staff and outpatient providers.

These statements, however, do not support the

proposed study topic and intervention.

To demonstrate that the study topic is relevant to

the local Medicaid population, GCBH should first

clarify whether the gap in service is due to the

availability of physical health information to

mental health providers and/or to the utilization of

that information by mental health providers in

mental health treatment planning. GCBH then

should provide evidence to support the study topic

selection.(e.g., data on the number of enrollees in

its population with chronic medical conditions,

evidence that information on these enrollees’

physical health is not being made available to

mental health providers, discussion of current

utilization of physical health information by

mental health providers, and an explanation of

how missing physical health information has had

a deleterious impact on local enrollee health

outcomes, status, or satisfaction).

In its discussion, GCBH used “care coordination”

and “integration” to define the study problem and

the RSN’s vision. GCBH described the link

between the two concepts as follows: “Effective

coordination of care is at the heart of matching

services and information to individual needs at the

right time and in the right place.” Again, GCBH

needs to discuss the current status of coordination

of services and information at the RSN’s provider

agencies and how coordination is falling short of

the RSN’s vision.

GCBH stated that discussions about the selection

of a PIP topic had occurred during meetings of its

Clinical Directors Committee, Quality

Management Oversight Committee, Board of

Directors, and Regional Advisory Board, which

includes enrollees and their family members. The

RSN did not discuss how it identified the study

topic or why it prioritized this topic over others.

GCBH noted that the topic of integration of

physical and mental health was aligned with state

contractual guidelines on integration and with the

RSN’s own practice guidelines for recovery.

To meet this standard fully, GCBH needs to

clarify the problem it plans to address, provide

data and other evidence demonstrating that an

opportunity for improvement exists for the local

Medicaid population; and describe the PIP

selection and prioritization in more detail.



Standard 2: Study Question Score: 80 (Substantially met)

To meet Standard 2, the RSN needs to present a

study question that provides a clear framework

for data collection, analysis, and interpretation.

The study question should refer to the proposed

intervention, a study population (denominator), a

measure (numerator), a metric (e.g., average,

percentage), and a direction of desired change.

GCBH presented this study question:

“Will the training of outpatient providers and the

integration of physical healthcare information into

mental health records lower PRISM scores for

high medical risk psychiatric inpatient

population?”

The study question includes an intervention

(training and integration of physical health

information), a study population (high medical

risk psychiatric inpatients) a numerator (PRISM

scores), and a direction of change (lower). To

compare results across measurement periods, the

question needs to incorporate a metric associated

with the numerator, e.g., average. Also, GCBH

should clarify that the outpatient providers are

mental health service providers.

The appropriateness of the numerator is discussed

under Standard 3.


include a metric in the study question and clarify

the term “outpatient provider.”

Standard 3: Study Indicator Score: 55 (Partially met)

To meet Standard 3, the RSN needs to define the

measure (numerator) and study population

(denominator); define key terms; and discuss the

basis for adopting the indicator as a valid proxy

for enrollee outcomes, satisfaction, or quality of

care.

GCBH identified the following indicator

elements.

Denominator or study population:

Medicaid eligible as determined by the

Authorization Center

18 years or older at the time of admission

(to an inpatient facility)

discharged from a community-based

inpatient psychiatric facility into a GCBH

outpatient program

had health information sent to enrollee’s

mental health outpatient team

had at least one outpatient appointment

high medical risk as evidenced by a

PRISM score ≥ 0.5

Numerator: PRISM scores for enrollees in the

denominator in each measurement period.

GCBH defined the terms adult, Authorization

Center, discharged, and Medicaid eligible.

Definitions of other key elements included:

Continuous enrollment: Enrolled in outpatient

services in each measurement period. The RSN

did not discuss any allowable gaps in enrollment.

GCBH provided this definition under numerator

inclusion criteria, but the criterion actually applies

to the study population.

Outpatient visit: At least one routine visit

following initial intake. In the onsite PIP

interview, GCBH explained that the enrollee’s

treatment plan (and incorporation of the PRISM

data) would be reviewed at a routine visit

following hospitalization. It is not clear that a

single outpatient visit allows enough time for a

mental health provider to incorporate physical

health needs into mental health treatment goals

and treatment (thereby reducing the PRISM

score). GCBH’s report should document its

rationale for this inclusion criterion.

Health information: A PDF printout of the

PRISM report submitted via the GCBH provider

portal.

PRISM: Predictive Risk Intelligence System. A

decision support tool used to support care

management interventions for high-risk Medicaid

enrollees.



PRISM score: Medical cost risk score calculated

from information from medical, social service,

behavioral health, and long term care systems. A

PRISM score is calculated “for each Medicaid

individual who has received medical and/or

psychiatric treatment and/or medications (and for

which P1 received a bill) within the last 24

months.” GCBH should discuss how often a

PRISM score is recalculated.

High medical risk: In the onsite PIP interview,

GCBH indicated that it would use PRISM scores

to identify “high risk medical” enrollees. The

RSN should document its reason for using a

PRISM score of ≥ 0.5 to indicate “high medical

risk,” and provide evidence that a high PRISM

score is always correlated with high medical risk,

i.e., it is not possible for an enrollee to have a

high PRISM score without having serious

physical health needs.

Community-based inpatient psychiatric

setting: GCBH should indicate the number of

facilities involved in the project.

The definition for outpatient team is missing from

the report.

In justifying its selection of the study indicator,

GCBH noted that PRISM reports and scores are

easily available to RSN staff and can be shared

confidentially with outpatient providers. GCBH

described PRISM as a decision support tool used

to identify those enrollees most in need of care

coordination and then to develop appropriate

interventions. In both its report and in the onsite

review, GCBH emphasized the feasibility of using

this study indicator. However, the RSN did not

provide evidence that PRISM scores are a valid

tool for assessing the effectiveness of care

coordination in the form integrating physical

health needs into mental health treatment plans,

over time. Since GCBH stated that the PRISM

score is calculated using data from multiple

systems over a two-year period, it is not clear that

a reduction in the PRISM score could necessarily

be attributed to the PIP intervention.

If GCBH cannot successfully address the multiple

challenges surrounding the use of average PRISM

scores as an indicator, the RSN should consider

selecting a different measure (e.g., the percent of

enrollees or high medical risk enrollees who have

mental health treatment goals that are consistent

with their PRISM scores).


define all key indicator elements and provide

evidence for the validity of its study indicator, or

select an appropriate study indicator.

Standard 4: Denominator (Study Population) Data Collection Score: 60 (Partially met)


inclusion and exclusion criteria for the study

population; document all data sources, including

fields, codes, and calculations; and describe data

validation procedures. If a sample is selected, the

RSN needs to describe the sampling methods.

GCBH described its study population as a pilot

study using a convenience sample. Instead of

covering the entire adult Medicaid population,

GCBH is targeting enrollees who have been

discharged from an inpatient facility. The RSN

stated that its authorization staff can identify high

medical risk enrollees through the in-house CIS,

and has easy access to the PRISM system. In the

onsite PIP interview, GCBH explained in detail the

reasons why it chose to focus on inpatient and high

medical risk enrollees. The RSN should document

this rationale in its discussion about sampling, and

discuss the possible effect a convenience sample

will have on study results and the generalizability

of the results to the larger RSN population.

Data on study population age, Medicaid

eligibility, and PRISM scores are available from

the Provider One database. Discharges from an

inpatient setting are available from the GCBH

Authorization Center and CIS database. GCBH

did not provide details about data source, codes,

and calculations for continuous enrollment criteria

and study population criteria listed under



Standard 3 (at least one outpatient appointment,

and dissemination of health information to

outpatient team).

The RSN listed no study denominator exclusions.

The RSN should consider whether exclusion

criteria may include situations in which enrollees

are not available for mental health treatment (e.g.,

incarceration, admission to a long-term care

facility, etc.).

The RSN’s quality manager will track the data on

an Excel spreadsheet. With regard to ensuring that

the study population was captured accurately,

GCBH stated that it used Provider One to validate

Medicaid eligibility data. The RSN did not

describe validation procedures for any other data

elements, including how it will validate correct

data entry of the enrollee demographics and

PRISM scores into the Excel spreadsheet and

receipt of health information by the mental health

provider.


ensure that inclusion criteria for the study

denominator are consistent between standards;

provide the source, calculations, and codes for all

inclusion criteria; describe validation procedures

for every data element; and provide more details

on the rationale for using a convenience study

sample.

Standard 5: Numerator (What Is Being Measured) Data Collection and Analysis Plan Score: 70 (Partially met)


inclusion and exclusion criteria for the numerator

(what is being measured); document all data

sources, including fields, codes, and calculations;

describe data validation procedures; and present

a clear data analysis plan, including time frames

for the measurement and intervention periods,

and an appropriate statistical test to measure

differences between the baseline and


The numerator for this study is represented by

PRISM scores on an individual enrollee’s PRISM

report generated from the website. There are no

true exclusion criteria.

GCBH authorization staff will send a PDF copy

of the PRISM report and scores to the quality

manager when an enrollee is discharged from an

inpatient facility. The quality manager will track

the baseline PRISM scores on an Excel

spreadsheet, and will “track and monitor” PRISM

scores for each remeasurement period. GCBH did

not discuss whether authorization staff would also

be responsible for generating remeasurement

PRISM scores.

Regarding validation procedures, GCBH noted:

“PRISM score accepted as valid.” Even though

the PRISM score is considered valid as reported,

GCBH needs to discuss how it will ensure that the

data entry into its own tracking system is accurate.

GCBH provided the following study timeline.

Baseline: January 1–June 30, 2014

Intervention: January 1–June 30, 2014

First remeasurement: July 1–December 31, 2014

Second remeasurement: January 1–June 30, 2015

GCBH plans to follow a single cohort of enrollees

who meet the inclusion criteria and are discharged

from an inpatient facility from January through

June 2014. Remeasurement will occur every six

months for each study enrollee according to their

discharge dates in the baseline measurement

period. GCBH did not describe an acceptable

window for the measurement periods, i.e., “5–7

months” for “6 months.” The intervention will

begin after an enrollee’s baseline PRISM report is

sent to the enrollee’s outpatient provider after

discharge. In this case, “baseline” is “the first

measure” measurement period.

GCBH will conduct comparisons with a paired

t-test for the first remeasurement and a repeated

measure ANOVA for the second remeasurement

(comparing three measurement periods), with a

probability level of p<.05.




describe how PRISM scores will be collected and

disseminated for the remeasurement periods,

describe the acceptable window for each of its

measurement periods, and discuss how it will

ensure that data entry of PRISM scores is

accurate.

Standard 6: Study Intervention Score: 45 (Minimally met)

To meet Standard 6, the RSN needs to select an

improvement strategy that will affect a wide range

of enrollees or a high-risk enrollee population,

and that is reasonably expected to result in

measurable improvement. The RSN needs to

discuss the basis for adopting the intervention;

document the implementation of the intervention,

including dates and locations of principal

activities; discuss cultural competence; and track

how effectively the intervention was implemented.

In the study question (Standard 2), GCBH

described the intervention as provider training and

the “integration of physical healthcare

information into mental health records.” Under

Standard 6, GCBH described its intervention

strategy as the dissemination of a PDF summary

of the PRISM report on study enrollees by

Authorization Center staff to the outpatient

mental health provider at the time of enrollee

discharge from an inpatient facility. In addition to

disseminating the PRISM report summary, GCBH

plans to conduct a 10- to 15-minute web training

for provider staff who will receive the document.

The training will focus on the ways the health

information in the PRISM report “can inform

mental health treatment planning.”

The assumption is that the provider, having

received both the information and the training,

will integrate an enrollee’s physical health issues

into the mental health treatment plan. However, in

the PIP interview, GCBH reported difficulty in

directing provider actions. Therefore, GCBH

should discuss how it will engage provider staff in

participating in training and changing behavior

(developing new treatment goals).

GCBH needs to provide more details about its

intervention, e.g., the type and presentation of

information in the PRISM report summary; the

allowable time frame for disseminating the report

to the provider; the number and roles of the staff

who will be trained; how newly hired staff would

receive the training; and whether or not the

training would be repeated. Training is expected

to begin in October and continue through

December 2013, followed by implementation of

the intervention in January 2014.

GCBH described the role of its Authorization

Center and discussed the feasibility of

implementing this intervention due to the

availability of qualified staff and their ability to

access PRISM reports. Although the intervention

may be feasible, GCBH still needs to discuss the

barriers upon which it based its intervention.

What barriers do mental health providers face in

integrating physical health information into

mental health treatment plans? How does the

receipt of information and a 15–minute training

address those barriers? GCBH also needs to

discuss why the intervention can be expected to

improve the study indicator; for example, because

it is an evidence-based practice or because its

successful implementation is assured by complete

support of all stakeholders.

In discussing its tracking and monitoring plan,

GCBH described the supervision of the

Authorization Center staff and stated that the

quality manager would track and monitor study

data. While it is important to establish that

qualified and supervised staff will be

disseminating information, GCBH needs to focus

on how it will track whether or not the

information (PRISM report) was disseminated

and received as planned, whether or not mental

health providers watched the training, and

whether or not mental health providers included

physical health issues into the mental health

treatment plans.

Regarding cultural and linguistic appropriateness,

GCBH stated that the web training will “include

and address cultural sensitivity as is it applies to



physical health integration into mental health

treatment.” The RSN should provide more details

about how the training addresses cultural issues.

Also, GCBH could discuss whether the PRISM

report takes cultural (e.g., disabilities, serious

mental illness) and linguistic (e.g., non-English

speakers) factors into account.


thoroughly describe its intervention strategy;

explain how the intervention addresses identified

barriers and can be expected to improve the study

indicator; describe how it will track the successful

implementation of the intervention; and provide

more details on how it will ensure the intervention

is culturally and linguistically appropriate.

Standard 7: Study Results Score: 0 (Not met)

To meet Standard 7, the RSN needs to present

results according to the data analysis plan,

including the study indicator, the original data

used to compute the indicator, and a statistical

test to measure differences between the baseline

and remeasurement periods; and discuss how the

intervention influenced the results.

GCBH did not report on this standard because no

results were available at the time of the PIP

review.

To meet this standard fully, GCBH needs to report

data for the baseline and remeasurement periods,

compare the indicator between the two periods

with an appropriate statistical test, and discuss

how the intervention influenced the study results.

Standard 8: Interpretation of Study Results Score: 0 (Not met)

To meet Standard 8, the RSN needs to assess

whether any reported improvement is “real” by

documenting that baseline and remeasurement

data were collected using the same methods and

are comparable; discuss the statistical and

clinical significance of the study results; address

barriers to improvement and lessons learned

during the PIP process; and identify confounding

factors that may have affected the results.

The project has not advanced far enough for

GCBH to address this standard.


discuss whether the intervention resulted in

statistically and clinically significant improvement;

identify confounding factors or barriers that had an

impact on study results and discuss how they were

addressed; and describe any lessons learned.

Standard 9: Study Modifications After the First Remeasurement Score: n.a.

To meet Standard 9, the RSN needs to document

modifications to the intervention, or added

interventions, planned or implemented after the

first remeasurement period; and discuss changes

in other aspects of the PIP based on lessons

learned from data analysis or barrier analysis.


the study had not progressed to a second

remeasurement.

Standard 10: Overall Study Results Score: n.a.

To meet Standard 10, the RSN needs to report

complete study results for two or more

measurement periods, including the study

indicator, original data used to compute the

indicator, and a statistical test of group

differences; and interpret the statistical and

clinical significance of the overall results, discuss

lessons learned, and determine if goals were met

and sustained improvement was achieved.


the study had not progressed to a second

remeasurement.

2013 Greater Columbia Behavioral Health EQR Report: Compliance review follow-up


COMPLIANCE REVIEW

FOLLOW-UP

Acumentra Health’s 2012 compliance review

addressed GCBH’s compliance with federal and

state standards related to eight major sections of

managed care operations. Each review section

contained elements corresponding to relevant

sections of 42 CFR §438, DBHR’s contract with

RSNs, the Washington Administrative Code, and

other state regulations where applicable. The

reviews followed a protocol adapted from the

CMS protocol and modified with DBHR’s

approval. The provisions of Washington’s

Medicaid waiver and the RSN contract are such

that some parts of the federal protocol do not

apply directly to RSNs.

In 2013, Acumentra Health reviewed GCBH’s

response to the specific 2012 EQR findings for

which DBHR required the RSN to perform

corrective action. Table 3 summarizes the results

of this follow-up review.

2013 Greater Columbia Behavioral Health EQR Report: Compliance review follow-up


Table 3. Results of compliance review follow-up for Greater Columbia Behavioral Health.

2012 findings 2012 recommendations RSN response

Delivery Network

GCBH does not track and monitor all requests for second opinions.

GCBH has corrected this finding, and current processes include tracking and monitoring. The RSN has reviewed the process with local clinical directors.

The RSN described a process that meets the requirement.

Coordination and Continuity of Care

None

Coverage and Authorization of Services

None

Provider Selection

GCBH does not have a mechanism in place to monitor the results of its credentialing and recredentialing process.

GCBH stated that it had a process to routinely track and monitor the credentialing and recredentialing process during 2012. In 2013, the RSN confirmed that this process includes routine monitoring of the Office of the Inspector General’s website to check each credentialed provider and staff person. Yearly, the RSN audits a sample of agency credentialing files onsite.


Subcontractual Relationships and Delegation

None

Practice Guidelines

GCBH does not review and update its practice guidelines to ensure that they still apply to enrollees needs and include any updated clinical recommendations.

GCBH has revamped its Quality Assurance program. The draft is to be vetted by the community advisory committee and the board. Practice guidelines have been updated and placed on the RSN’s website.


QA/PI Program

GCBH lacks mechanisms to detect, identify, and monitor for over- and underutilization of services.

GCBH has reviewed local data and performance measures to identify over/underutilization. The RSN has added three additional mechanisms for monitoring the service penetration rate, recidivism of inpatient stays for children, and information from the PRISM database to identify overutilization. This year, the RSN will determine the baseline experience.


Certification and Program Integrity

None

2013 Greater Columbia Behavioral Health EQR Report: ISCA Overview


INFORMATION SYSTEMS

CAPABILITIES ASSESSMENT

Acumentra Health examined GCBH’s

information systems and data processing and

reporting procedures to determine the extent to

which they supported the production of valid and

reliable state performance measures and the

capacity to manage care of RSN enrollees.

The ISCA procedures were based on the CMS

protocol for this activity, as adapted for the

Washington RSNs with DBHR’s approval. In

2013, Acumentra Health added a new review

subsection, Meaningful Use of Electronic Health

Records, per the CMS protocol published in

September 2012. Due to the timing of the CMS

waiver, this section will be reviewed but not

scored in 2013.

The 2011–2013 RSN contracts include Exhibit C

with more detailed data security requirements.

These criteria were included in the 2013 ISCA for

the first time.

Acumentra Health interviewed four contracted

provider agencies to gain understanding of the

flow of encounter data from providers to GCBH.

These interviews targeted the agencies’ data

validation processes; diagnoses and procedure

codes captured in their billing systems; handling

of Medicaid/Medicare dual enrollees; types of

encounter data forwarded to GCBH; and methods

for submitting encounter data to GCBH.

The ISCA review was organized in two main

sections—(1) Data Processing Procedures and

Personnel and (2) Data Acquisition Capabilities—

with nine subsections, each containing review

elements from the ISCA protocol. Appendix B

explains the review criteria in greater detail.

Within each subsection, Acumentra Health used

the information collected in the ISCA data

collection tool, responses to interview questions,

and results from the security walkthrough to score

the RSN’s performance on each element on a

scale from 1 to 3 (see Table 4).

After scoring the individual elements, Acumentra

Health combined the scores and calculated a

weighted average score for each subsection. The

detailed criteria for scoring are available from

Acumentra Health upon request.

Table 4. Scoring scheme for ISCA standards.

Score Rating Definition

2.6–3.0 Fully met (pass) Meets or exceeds the standard requirements.

2.0–2.5 Partially met (pass) Meets essential requirements of the standard but is deficient in some areas.

< 2.0 Not met (fail) Does not meet the essential requirements of the standard.

– N/A Not applicable.

2013 Greater Columbia Behavioral Health EQR Report: ISCA Overview


Summary of review results

During the review year (January–December

2012), GCBH used a Microsoft SQL database

management system to process encounter data in

house. In October 2012, GCBH replaced the 2005

version with MS SQL Server 2012. The RSN’s

contracted provider agencies used various practice

management systems to collect, process, and

submit encounter data.

At the end of 2012, GCBH began processing

outpatient and inpatient service authorizations in

house, assuming a function previously performed

by an outside contractor.

Since the previous full ISCA in 2011, GCBH has

experienced considerable turnover in IT staffing.

Standard upkeep and maintenance have stalled in

this period of transition, and issues from the

previous ISCA, IT practices, and policies have not

been addressed.

Acumentra Health’s review found that in 2012,

GCBH partially met federal standards related to

data processing procedures and personnel, and

fully met data acquisition capabilities standards.

Table 5 summarizes the ISCA section scores and

ratings.

Table 5. Weighted average scores and ratings on ISCA sections.

Review section/subsection Score Compliance rating

Section 1: Data Processing Procedures and Personnel

A. Information Systems 2.3 Partially met

B. Staffing 2.2 Partially met

C. Hardware Systems 2.6 Fully met

D. Security 1.5 Not met

Section 2: Data Acquisition Capabilities

A. Administrative Data (claims and encounter data) 2.9 Fully met

B. Enrollment Systems (Medicaid eligibility) 2.7 Fully met

C. Vendor Data Integration 2.2 Partially met

D. Provider Data (compensation and profiles) 2.3 Partially met

E. Meaningful Use of Electronic Health Records n.a. n.a.

2013 Greater Columbia Behavioral Health EQR Report: Data Processing Procedures and Personnel


ISCA Section 1: Data Processing Procedures and Personnel

Section 1A: Information Systems Score: 2.3 (Partially met)

This subsection reviews the RSN’s systems

development life cycle (SDLC) and supporting

environments, including database management

systems and/or billing software, programming

languages, and training for programmers.

GCBH transitioned in October 2012 from

Microsoft SQL Server 2005 to Microsoft SQL

Server 2012 for Medicaid encounter data

processing, data analysis, and reporting.

GCBH’s customized Microsoft SQL database is

maintained by the contracted consultant who

developed it. The RSN has no trained backup staff

who could take over if the consultant became

unavailable. GCBH has created documentation for

the database, but has not finished developing

operating procedures or training for staff.

GCBH has employed several IS managers since

the previous ISCA. Many IT practices, issues, and

policies have gone unaddressed during this period.

At the time of the 2013 review, the current IS

manager had been employed for about one

calendar quarter.

Recommendations

GCBH lacks a formal, written quality assurance

(QA) process for its information systems for

encounter data processing, analysis, and reporting.

GCBH stated that it has developed a QA process,

but provided no documentation.

GCBH needs to implement a formal,

written QA process for encounter data

processing, analysis, and reporting.

GCBH relies solely on a contracted consultant for

its programming. The RSN does not assign internal

staff to perform programming or to monitor the

code produced by the consultant. The absence of

backup, coupled with incomplete documentation

and training materials, creates a risk of unplanned

and extended disruptions of GCBH’s internal

processes and claims processing.

GCBH needs to develop a formal process

for monitoring of outsourced IT services.

GCBH needs to designate a backup

administrator who could keep the

customized database running if the current

consultant became unavailable.

GCBH does not have formal contracts or business

associate agreements for outsourced IT services.

GCBH needs to formalize its relationships

with providers of outsourced IT services.

Written agreements should specify the

consultant’s roles and responsibilities as

well as any deliverable requirements.

GCBH appears to lack appropriate documentation

to enable staff to understand RSN systems and

business processes.

GCBH needs to develop documentation of

RSN systems and business processes for

the benefit of staff.

GCBH’s computer programming and data report

production processes lack a formal peer review

process to validate the accuracy and completeness

of data before production.

GCBH needs to develop a formal peer

review process for internal programming

and report production.

GCBH lacks a formal SDLC and sign-off process

for code changes to RSN systems.

GCBH needs to develop a formal process

for software development changes,

including peer review, testing, and

authorization for production.

GCBH does not use a formal version control

process for report development.

GCBH should consider implementing a

formal version control process for its

reporting and archiving process.



GCBH does not use version control management

software for internal reporting. Instead, GCBH

staff renames previous versions of files or

programs, and edits new copy as needed, keeping

previous versions for reference.

GCBH should consider using version

control software to enable a more robust,

quick, and efficient process that is less

prone to error.

Section 1B: Staffing Score: 2.2 (Partially met)

This subsection assesses physical access by the

RSN’s staff to IT assets, as well as specific

training requirements for claims processing and

care authorization staff, and RSN staffing.

GCBH performs authorizations for inpatient and

outpatient mental health services through its

customized in-house CIS. A credentialed

utilization management (UM) specialist performs

all authorization decisions on the basis of Access

to Care standards and other information

submitted by the provider agency.

The review revealed a lack of clarity about team

members’ individual roles and responsibilities, as

well as about backup coverage. Processes were

informal, and team members appeared to operate

in silos, creating potential single points of failure

in their respective areas of expertise.

Staff appeared to lack understanding of many

aspects of the CIS database functionality, which

only the contracted consultant could address.

Many of the RSN’s business processes were

known only to the staff members handling those

responsibilities, and little or no documentation

appeared to exist.

Strengths

GCBH’s credentialed UM specialists

understand locally available resources.

Recommendations

Roles and responsibilities of individual GCBH

staff members appeared unclear.

GCBH needs to clearly define roles and

responsibilities of internal staff and of its

IT consultant.

GCBH needs to continue to monitor its IT

staffing to address underlying concerns,

and train existing staff to provide backup

for key activities.

Section 1C: Hardware Systems Score: 2.6 (Fully met)

This subsection assesses the RSN’s network

infrastructure and hardware systems.

GCBH’s CIS servers reside in the GCBH office

data center. These servers are under warranty until

second quarter 2014. GCBH’s previous IS

manager purchased a new server to implement a

virtualized CIS, but implementation was never

completed. GCBH plans to revisit implementation

plans for this new hardware solution.

Strengths

Servers are housed in a secure location

away from personnel who are not

authorized to have physical access.

GCBH takes advantage of redundant

software and hardware designs.

Recommendations

GCBH’s servers are approaching end of life and

need to be replaced. The new hardware purchased

nearly two years ago has gone unused and may be

nearing the end of its warranty period.



GCBH needs to pursue it plans to upgrade

to new equipment before the current

servers reach end of life.

GCBH needs to develop and implement

a plan to install or replace purchased

hardware in a manner that meets business

requirements. This hardware has a limited

warranty period and life expectancy, and

likely would need to be replaced soon after

being installed.

Section 1D: Security Score: 1.5 (Not met)

This subsection assesses the RSN’s information

systems for integrity and the ability to prevent

data loss and corruption. A security walkthrough

of the computer area and/or data center assesses

the possibility of a breach in security measures.

GCBH staff performs daily data backups to a

tape-based storage system. Encrypted backup

tapes are transported to a secure offsite location

once a week.

GCBH’s new IS manager has begun updating the

RSN’s Business Continuity/Disaster Recovery

(BC/DR) plan to meet current DBHR contract

requirements. GCBH’s provider agencies reported

that they maintain current disaster recovery plans.

Strengths

All RSN tapes containing protected health

information (PHI) are encrypted.

Findings

RSN Contract Exhibit C. GCBH’s security

policies and procedures have not been updated

recently and do not match current DBHR contract

requirements.

GCBH needs to continue updating its

security policies and procedures to align

with DBHR contract requirements, and

review the policies and procedures at least

annually.

RSN Contract Exhibit C – Data Security

Requirements. 1.b – Encrypting any data that

will be in transit outside the State Governmental

Network (SGN) or contractor’s internal network. At the time of the ISCA review, GCBH had at

least one laptop that was not encrypted. GCBH

reported that the RSN does not permit sensitive

data to be stored on a portable system until the

system has been encrypted. However, reliance on

a manual process by users to properly identify

PHI and avoid storage on unencrypted laptops

should be discouraged. GCBH reported that this

laptop has been encrypted since the site review.

The RSN is identifying encryption solutions for

all portable devices.

GCBH needs to proceed with its plans to

encrypt all laptops and portable devices to

align with DBHR contract requirements.

RSN Contract Exhibit C – Data Security

Requirements. 2.f. GCBH’s practice is to notify

DSHS within five days when an authorized

employee leaves the RSN or is terminated and thus

no longer requires SGN access. This practice does

not align with DBHR’s contractual requirement for

immediate notification.

GCBH needs to notify DSHS immediately

when an authorized employee or contractor

no longer requires access to the SGN.

GCBH’s policy to set devices to lock

automatically after 30 minutes of inactivity does

not align with DBHR contract requirements.

GCBH needs to revise its policy to set

devices to lock automatically after at most

20 minutes of inactivity.

GCBH’s provider agencies use various encryption

strategies for laptop computers, hard drives, and

other portable devices. One agency reported an

insufficient key length. Three agencies reported

that they do not encrypt backups.



GCBH should review provider agencies’

encryption strategies to ensure that they

are aligned with DBHR contract

requirements, current industry standards,

and HIPAA requirements.

Two provider agencies reported that passwords

used to connect to their networks or encounter data

systems do not require a complexity component.

One agency reported that its passwords expire

every six months.

GCBH needs to work with provider

agencies to ensure that passwords used to

connect to encounter data systems have

unique user IDs, sufficient complexity,

and forced change requirements that align

with DBHR contract requirements and

current industry standards.

RSN Contract 11.2.2: Business Continuity and

Disaster Recovery. GCBH’s BC/DR plan is not

aligned with current contractual requirements,

though the RSN has begun updating the plan.

GCBH needs to continue to align its

BC/DR plan with current contractual

requirements.

RSN Contract 11.2.2: Testing of DR/BC Plan.

GCBH has not tested its BC/DR plan recently.

GCBH needs to establish annual testing of

its BC/DR plan to meet DBHR contract

requirements.

Recommendations

GCBH transports backup tapes containing PHI to

an offsite location in a private vehicle. Although

the tapes are encrypted, this transportation method

is not an industry best practice.

GCBH has an older keypad entry system, used by

about 25 employees. All staff members use the

same entry code, which is changed at the time of

staff departures. This system cannot track individual

access, and is vulnerable to the risk of people

outside of the RSN learning the code.

GCBH should begin planning for a system

with more granular access control and

monitoring capabilities.

2013 Greater Columbia Behavioral Health EQR Report: Data Acquisition Capabilities


ISCA Section 2: Data Acquisition Capabilities

Section 2A: Administrative Data Score: 2.9 (Fully met)

This subsection reviews the RSN’s submission of

accurate information, process for describing

differences when verifying accuracy of submitted

claims, and data assessment and retention.

GCBH accepts encounter data from its provider

agencies in electronic format. During processing,

encounter data submissions run through an

automated, rules-based edit system to screen the

data, identify potential input errors, and ensure

compliance with the state’s Data Dictionary and

Service Encounter Reporting Instructions (SERI).

Screened encounter data submissions are

converted to the HIPAA-compliant 837 format

before being transmitted to DBHR via a secure

shell connection. As required by DBHR, GCBH

verifies and certifies batched encounter data for

accuracy and completeness before transmitting

data. GCBH’s director is responsible for ensuring

compliance with state reporting requirements.

Strengths

GCBH has been able to meet DBHR

requirements for submission of encounter

data even with staff turnover issues.

GCBH sends encounter data to the state on

a weekly basis.

Recommendations

GCBH lacks a process to link authorization data

with encounters. As a result, encounters may be

accepted and processed without valid authorization

or reauthorization.

GCBH needs to integrate its authorization

and encounter processes. This should

include clearly defining the process that is

followed if an encounter is submitted to the

RSN for a service without an authorization,

and ensuring that procedures are in place to

prevent the delivery of services that have

not been authorized.

It is unclear whether agencies are capturing and

reporting only the diagnosis assigned at the time of

the enrollee’s intake assessment or reassessment.

The diagnosis at the time of the service encounter

may not reflect the current diagnosis.

GCBH needs to develop a method to

ensure that the diagnosis being treated at

the time of service is reported on the 837.

Section 2B: Enrollment Systems Score: 2.7 (Fully met)

This subsection assesses the RSN’s systems

pertaining to Medicaid enrollment and

disenrollment, tracking claims and encounter data,

Medicaid enrollment data updates, Medicaid

enrollment code, and data verification.

GCBH relies on its provider agencies to check

enrollee eligibility at each service encounter. One

agency does not check eligibility at each service

encounter, but checks eligibility monthly. GCBH

staff verifies eligibility at the time of service

authorization or reauthorization.

GCBH loads the state eligibility data into its CIS

on a regular basis, but it is unclear how the RSN

uses the eligibility data.

Findings

RSN Contract 4.7.1: The Contractor must verify

eligibility for Title XIX prior to the provision of

non-crisis services to an enrollee. GCBH relies

on providers to verify eligibility at the time of

service, and does not perform eligibility checks on

encounter data.

2013 Greater Columbia Behavioral Health EQR Report: Data Acquisition Capabilities


GCBH needs to ensure that eligibility

checks occur before submitting encounter

data to the state.

GCBH needs to work with providers to

ensure that they verify enrollment at each

service encounter.

Section 2C: Vendor Data Integration Score: 2.2 (Partially met)

This subsection assesses how the RSN integrates

vendor data with administrative data for data

completeness and quality.

GCBH performs encounter data validation audits

yearly. The proportionate number and sampling of

encounters to be audited for the year is based on

the previous contract year’s ratio of services by

provider agency, program/project code, and

enrollee age category (children age 0–17 and

adults age 18+).

Strengths

GCBH’s provider agencies met contract

requirements for EDV standards.

GCBH has worked with agencies to

address issues that need correction but do

not require a formal corrective action plan.

Recommendations

One provider agency reported that some changes

to encounter data are made by IT staff, and that no

clinical review occurs to verify the completeness

and accuracy of those changes. Some data

changes, such as in progress notes, should be

made only by the rendering practitioner or by an

appropriate clinical supervisor.

GCBH needs to ensure that provider

agencies understand which data changes

need to be verified by clinical staff, and

that agencies implement an appropriate

review process for data changes.

One provider agency reported that it submits

current data on the 1st and the 15

th day of each

month. GCBH must wait until these dates to

obtain the most accurate information. Thus, when

changes to encounter data occur, GCBH’s CIS

data may not be kept current with information in

the providers’ data systems.


agencies to ensure that data changes are

captured in both the RSN and provider

agency data systems.

One provider agency reported difficulty in

interpreting the manual that is intended to help

agencies structure the data they send to GCBH.

According to this agency, the manual’s ambiguity

causes the agency to make several changes to its

systems for a single requirement or field.


agencies to develop a manual with the

details necessary to develop data files

appropriately with little intervention. This

process could include a work group to

determine the level of detail the agencies

need in order to develop these files.

2013 Greater Columbia Behavioral Health EQR Report: Information Systems Capabilities Assessment


Section 2D: Provider Data Score: 2.3 (Partially met)

This subsection assesses how the RSN maintains

its provider directory, as well as the RSN’s fee

schedules and contractual payment updates.

GCBH stated that it no longer monitors for

missing encounters as part of EDV activities.

RSN staff members perform limited monitoring

and reporting of the number of encounters

received or expected.

Recommendations

GCBH has an informal process for updating its

provider directories.

GCBH should formally define its process

to update provider directories, including

steps for adding providers, approval of

credentialing, and updates to the CIS to

identify that providers are now authorized

to serve enrollees.

GCBH does not appear to distinguish between

services paid for by Medicaid and those paid by

other funding streams.

GCBH needs to establish appropriate

policies and procedures to distinguish

services paid for by Medicaid vs. those

funded by other payment sources.

Section 2E: Meaningful Use of Electronic Health Records Score: n.a.

This subsection assesses how the RSN and its

contracted providers use electronic health records

(EHRs). Due to the timing of the CMS waiver,

Acumentra Health and DBHR determined that

this section would be reviewed in 2013, but no

scores will be reported this year.

Recommendations

GCBH should develop EHR-related

policies and procedures in advance of

implementation, specifying the RSN’s role

in EHR adoption, expectations during

implementation, and plans for transition

periods when data may not be available.

GCBH should consider testing with

provider data systems during provider

agency EHR implementation.

GCBH should consider monitoring data for

quality, completeness, and accuracy

throughout EHR implementation, including

a post-implementation review.

2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation


ENCOUNTER DATA VALIDATION

Medicaid encounter data must be complete and

accurate to be useful in calculating statewide

performance measures and determining managed

care capitation rates. DBHR’s contract requires

each RSN to conduct an annual encounter data

validation (EDV) to determine the accuracy of

encounter data submitted by providers.

As an independent check of the RSNs’ EDV

results, Acumentra Health audits and verifies the

EDV process for each RSN. DBHR accepts the

RSNs’ self-validation of their encounter data,

subject to audit and verification by Acumentra

Health.

For each RSN, the 2013 EDV involved:

checking each field in the state’s

outpatient electronic data set for missing

and out-of-range data and logic problems

comparing specific data fields in clinical

records of the RSN’s providers against the

state’s electronic data sets to determine

whether data submitted by the providers

were accurate and complete

As a special topic, the 2013 EDV also examined

the degree to which providers’ clinical records

demonstrated adherence to the “golden thread”

of mental health therapy:

1. Does the assessment in the clinical record

substantiate the individual’s diagnosis?

2. Are the documented goals of the treatment

plan consistent with the diagnosis?

3. Do the progress notes address the

individual’s progress toward meeting the

treatment plan goals?

Appendix C presents an overview of Acumentra

Health’s EDV procedures.

Greater Columbia Behavioral Health’s EDV procedure

According to documentation submitted for this

audit, GCBH validated encounter data submitted

by provider agencies to the RSN against the chart

notes during reviews of six provider agencies.

GCBH performs annual EDV activities at a

selection of agencies and creates agency-specific

analyses to summarize the results. The RSN

samples from all encounters that are expected to

be sent to the state. The time frame for the

sample reviewed was six months prior to the

review for each agency, so the time frame was

slightly different for each.

Acumentra Health recommends that for all EDV

activities, the RSN use encounters processed by

the state rather than encounters sent to the state.

That way, the RSN can ensure that its encounter

data are received and processed as expected, and

the RSN will have the opportunity to address any

data errors in a timely manner.

This review first assesses GCBH’s sampling

procedure, then discusses the strengths of the

RSN’s data entry tool and analytical procedures.

Sampling procedure

To evaluate the RSN’s sampling methodology,

Acumentra Health examined the overall sample

size and whether the sample was chosen in a

random process.

GCBH reviewed encounters at six provider

agencies. Six other agencies were excluded from

this annual review due to exemplary review

outcomes in last year’s review, and a seventh

agency was excluded from review due to its

departure from the RSN.

GCBH reviewed 1,126 encounters in 272 charts

for Medicaid enrollees, far more than required by

the DBHR contract.

To be selected in the review sample, enrollees

must have received at least six services within six

months and have received an intake, medication

monitoring/management, and/or crisis service.



GCBH’s quality manager selected the charts to be

reviewed in a manual random and blind fashion,

lacking information other than age, gender, and

enrollee ID. The sample was selected in

proportion to the age and gender distribution of all

enrollees served by each agency, and sample size

for each agency matched the proportion of clients

served in the RSN.

Acumentra Health recommended in 2012 that

GCBH develop a computerized approach to select

a random sample. A robust computerized random

sampling would:

1. be more efficient in choosing the sample

in a much shorter period of time

2. select chart and encounters over the entire

list of possible choices, not just up to the

point of reaching target numbers and ratios

of gender and age distribution

3. result in a different list of enrollees and

encounters each time the sampling

procedure is run on the same set of data

While the manual procedure may generate a

random sample, a computerized algorithm would

ensure that the three criteria above are met each

time the program is called. Using an appropriate

computerized approach would ensure that each

chart and encounter has an equal probability of

being chosen.

GCBH’s procedure is partially adequate for

choosing an EDV sample. The sample meets the

size requirement, but the random sampling

procedure should be based on a computerized

rather than manual approach.

Note: Following the EDV review, which was

based on a December 2012 report by the RSN,

GCBH reported that it had implemented a

computerized sampling approach in 2013.

Record review tools, process, and results

GCBH uses an Access database to record its

EDV results. The encounters selected by the

random sampling procedure are loaded into the

database, and results are collected on a single

form where the reviewer marks any item that

does not match the chart. After the review is

complete, the data are pulled out of the Access

database and into MS Excel for analysis.

Acumentra Health repeats the recommendation

from 2012 that GCBH improve its data entry tool

to perform the calculations now performed with

Excel. There appears to be a transfer of data back

and forth between Access and Excel. Data are

exported to Excel for purposes of sampling, then

the sample is moved back into Access so results

can be collected, then results are analyzed in

Excel. It would be more efficient to keep the data

in Access and analyze the results using Access

reports that can show agency-level as well as

RSN-level summary statistics.

GCBH reviewed all elements that are required to

be reviewed per the DBHR contract except

“service code agrees with treatment described.”

Acumentra Health recommends reviewing this

element in future EDV activities. GCBH also

reviewed additional demographic elements

(gender, date of birth, ethnicity, Hispanic origin,

language, and sexual orientation) and the “golden

thread” by examining whether the documentation

in the enrollee’s chart was sufficient to

demonstrate medical necessity.

One reviewer performs all EDV reviews and

analysis. However, it is unclear whether GCBH

reviews previous years’ procedures or uses an intra-

rater agreement test to ensure that the reviewer’s

scores remain consistent across agencies and across

time. If this process is not in place, Acumentra

Health recommends implementing it.

Reviewers examined records for Match, No

Match–Erroneous, No Match–Unsubstantiated,

and Insufficient Note. The last item is an RSN-

added element. No Match–Missing was not

reviewed. Acumentra Health recommends

examining charts for “missing” encounters, where

the clinical record contains evidence of a service

not represented in the electronic encounter data.

EDV results are presented at the data element

level, stratified by agency and aggregated by



error type for the RSN. None of the agencies

reviewed had an error rate that required a

corrective action plan.

GCBH’s record review procedure is adequate for

assessing the accuracy and completeness of the

EDV data.

Validation results

This report presents the EDV results in three

parts: first, the results of electronic data checks;

second, the results of comparing the clinical chart

documentation with the state’s electronic data, as

part of the onsite review; and finally, the results of

the “golden thread” analysis.

Electronic data checks

Acumentra Health analysts checked data fields in

329,813 outpatient encounters for missing and

out-of-range data and logic problems,

representing all outpatient encounters reported by

GCBH during October 2011–September 2012.

The fields examined included RSN ID, consumer

ID, agency ID, primary diagnosis, service date

and location, provider type, procedure code, claim

number, and minutes of service.

All fields were 100% complete and within

expected limits, although 150 claims (0.05%) had

a duplicate Claim ID.

Next, analysts checked the demographic data set,

examining 19,193 records. The fields examined

included RSN ID, consumer ID, first and last

names, date of birth, gender, ethnicity, Hispanic

origin, language preference, Social Security

number (SSN), and sexual orientation.

Considering mandatory fields, analysts found

2,526 records (13.2%) with out-of-range ethnicity

information, indicating that an invalid code was

submitted. Considering optional fields, analysts

found that 4,047 records (21.1%) omitted the

SSN. All other mandatory and optional fields

were 100% complete and within expected limits.

Table 6 summarizes the results of electronic data

checks for the outpatient and demographic data

sets.

Onsite review results

Acumentra Health staff audited 394 encounter

records for GCBH. The encounters were reported

in 107 charts. The data fields compared for each

encounter included procedure code, provider type,

service minutes, service date, and service location.

Acumentra Health staff reviewed the encounter

notes to verify that the procedure code accurately

described the treatment provided, and compared

electronic data from the state’s demographic data

set with the chart documentation for the 107

enrollees. Demographic fields that were compared

included first name, last name, date of birth,

ethnicity, and language.

The choices available to the audit team in

comparing electronic data with the source chart

documentation for each field were:

1. Chart matches state data

2. Data in chart missing from state data

3. Missing from both chart and state data

4. Could not locate in chart

5. Data found in chart do not match state data

Reviewers also compared encounters documented

in the clinical record with encounters in the

electronic data to identify “missing” encounters

that were documented in the clinical record but

were not sent to the state.

Table 7 shows the results of Acumentra Health’s

validation activity.

Within the demographic dataset, the chart

information matched the state data in 99.1% of

records for first and last name, and matched in

100% of records for date of birth. Ethnicity data

found in the chart matched the state data in 83.2%

of records, and language data found in the chart

matched the state data in 91.6% of records.

Of 394 encounters reviewed from the outpatient

data set, 79.6% had a service code that matched

the service described in the chart note. Provider

type, service location, procedure code, and service

date matched in 80% to 93% of encounters

reviewed, while only 38.6% of encounters had

matching data on minutes of service.



Table 6. Results of 2013 electronic data checks.

Field State standard % completea

Outpatient encounter data

RSN ID 100% complete (non-missing values), with values known to DBHR 100.0

Consumer ID 100% complete (non-missing values), with values known to DBHR 100.0

Agency ID 100% complete (non-missing values) 100.0

Primary diagnosis 100% complete (non-missing values), one diagnosis must be present 100.0

Service date 100% complete (non-missing values), must be in valid date format 100.0

Service location 100% complete (non-missing values), with values specified in data dictionary 100.0

Provider type 100% complete (non-missing values), with values specified in data dictionary 100.0

Procedure code 100% complete (non-missing values), with values specified in service instructions 100.0

Claim number 100% complete (non-missing values) 100.0

Minutes of service 100% complete for records with no per diem CPT/HCPCS codes 100.0

Demographic data

RSN ID 100% complete (non-missing values), with values known to DBHR 100.0

Consumer ID 100% complete (non-missing values) 100.0

First name 100% complete (non-missing values) 100.0

Last name 100% complete (non-missing values) 100.0

Date of birth Optional per the state’s Data Dictionary 100.0

Gender Optional per the state’s Data Dictionary 100.0

Ethnicity 100% complete (non-missing values), with values specified in data dictionary 86.8

Hispanic origin 100% complete (non-missing values) 100.0

Language preference 100% complete (non-missing values) 100.0

Social Security number Optional per the state’s Data Dictionary 78.9

Sexual orientation 100% complete (non-missing values) 100.0

aDue to rounding, some fields showing 100.0 percent completeness may have had a small number of missing data values.



There is a systematic issue converting service

minutes from the RSN to units required for

reporting to Provider One, which converts units

back to minutes in the data housed by DBHR.

Staff from DBHR are working on this issue.

In May 2012, the only month reviewed,

Acumentra Health reviewers found no “missing”

encounters.

“Golden thread” analysis

This portion of the EDV examines whether the

enrollee’s assessment substantiates the diagnosis,

whether the treatment plan is consistent with the

diagnosis, and whether progress notes address the

treatment plan.

The assessment substantiated the Category A

diagnosis in 89.5% of children’s charts and 89.7%

of adult charts. An additional 5.3% of children’s

charts and 3.4% of adults’ charts contained an

assessment that partially substantiated the

Category A diagnosis.

The Category B diagnosis, when applicable, was

substantiated in 100% of children’s charts. No

adult charts contained a Category B diagnosis.

The treatment plan was consistent with the

diagnosis and assessment in 95.1% of children’s

charts and 67.2% of adults’ charts, while the

treatment plan objectives were individualized in

95.0% of children’s charts and 82.0% of adults’

charts. The progress notes addressed interventions

identified in the treatment plan in 92.9% of

children’s charts and 78.3% of adults’ charts.

An additional 3.3% of adults’ charts partially


plan.

Table 8 displays the results of the golden thread

analysis for GCBH. “Not applicable” or blank

responses were removed from the table, so the

denominator for both “% Yes” and “% Partially”

includes only charts that were scored “yes,” “no,”

or “partially.”

Table 7. Results of Acumentra Health’s encounter data validation for Greater Columbia Behavioral Health.

Field Chart matches

state data

Data in chart missing from

state data

Missing from both chart and

state data

Data could not be located

in chart

Data found in chart do not

match state data

Demographic information from each clinical record reviewed (N=107)

First name 106 (99.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 1 (0.9%)

Last name 106 (99.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 1 (0.9%)

Date of birth 107 (100.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%)

Ethnicity 89 (83.2%) 0 (0.0%) 0 (0.0%) 2 (1.9%) 16 (14.9%)

Language 98 (91.6%) 0 (0.0%) 0 (0.0%) 2 (1.9%) 7 (6.5%)

Results from multiple encounters and a mix of services (N=394)

Provider type 353 (89.6%) 0 (0.0%) 0 (0.0%) 36 (9.1%) 5 (1.3%)

Minutes of service 152 (38.6%) 2 (0.5%) 0 (0.0%) 51 (12.9%) 189 (48.0%)

Service location 354 (89.9%) 0 (0.0%) 0 (0.0%) 34 (8.6%) 6 (1.5%)

Procedure code 316 (80.2%) 0 (0.0%) 0 (0.0%) 67 (17.0%) 11 (2.8%)

Service date 365 (92.9%) 0 (0.0%) 0 (0.0%) 28 (7.1%) 0 (0.0%)



Table 8. Results of “golden thread” analysis for Greater Columbia Behavioral Health.

Assessment substantiates the diagnosis; treatment plan is consistent with the diagnosis; progress notes address the treatment plan

Number of charts reviewed (N=102)

Children Adults Total

% “Yes” % “Partially” % “Yes” % “Partially” % “Yes” % “Partially”

1. Does the assessment substantiate the Category A diagnosis (if applicable)?

34 (89.5%) 2 (5.3%) 52 (89.7%) 2 (3.4%) 86 (89.6%) 4 (4.2%)

2. Does the assessment substantiate the Category B diagnosis (if applicable)?

4 (100.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 4 (100.0%) 0 (0.0%)

3. Does the treatment plan include interventions and goals consistent with issues identified in the assessment?

39 (95.1%) 0 (0.0%) 41 (67.2%) 0 (0.0%) 80 (78.4%) 0 (0.0%)

4. Do the progress notes address interventions identified in the treatment plan and the individual’s progress toward meeting stated goals?

39 (92.9%) 0 (0.0%) 47 (78.3%) 2 (3.3%) 86 (84.3%) 2 (2.0%)

5. Are the treatment plan objectives individualized? 38 (95.0%) 0 (0.0%) 50 (82.0%) 0 (0.0%) 88 (87.1%) 0 (0.0%)

NOTE: Proportions exclude “not applicable” or missing responses, so the denominator of each item may vary.



Discussion and recommendations

Acumentra Health finds GCBH’s EDV sampling

procedure partially adequate. The sample was an

appropriate size, but the randomization process

was manual, and Acumentra Health recommends

that GCBH develop a computerized algorithm to

select a random sample of encounters, for the

reasons noted above. The record review procedure

was adequate for assessing the accuracy and

completeness of the EDV data. We recommend

that GCBH develop the database data entry tool to

include reports that can show the summary

statistics presently calculated in Excel.

The quality of the electronic data file from DBHR

was generally good, except that 13.2% of records

contained out-of-range ethnicity data. Acumentra

Health recommends that GCBH determine how

150 claim IDs were duplicated so this type of

error can be prevented in the future.

Acumentra Health compared the above EDV

results with the results of the 2010 and 2008

encounter data validation. Among demographic

variables, there was an increase in the percentage

of records that matched the chart data in the date

of birth field, but a decrease in the percentage of

records that matched in the ethnicity field. There

was no change in the match rate for first and last

name, while language was not assessed in

previous years. Among the encounter data

variables, there was a decrease in the percentage

of records that matched the chart data in the

provider type, minutes of service, procedure code,

and service code matching the service described

in the chart note. There was no change in the

match rate for service location, while service date

was not assessed in previous years.

Golden Thread analysis

Assessments. Most mental health assessments

were well written for both children and adults.

They included documentation of the presenting

problem, medical necessity for treatment, clinical

formulation for diagnosis, and services.

Treatment plans. Although the majority of the

children’s treatment plans included interventions

and goals consistent with issues identified in the

assessment, only 67% of the adult treatment plans

included interventions, services, and measurable

goals that were consistent with the mental health

assessment.

GCBH needs to provide guidance to the

agencies to ensure that treatment plans

specify measurable goals, expected

outcomes, service(s) indicated, and

specific interventions consistent with the

enrollee’s assessment.

Progress notes. The progress notes for children

were consistent and well written. They typically


plan and the child’s progress toward meeting the

stated goals. However, as with the treatment

plans, the adult progress notes showed room for

improvement, as they did not consistently address

interventions identified in the treatment plan and

progress toward meeting the stated goals.

GCBH needs to provide direction to the

agencies to ensure that progress notes

clearly document the goals, treatment

interventions, enrollee’s response to the

interventions, and progress toward

meeting the goals.

2013 Greater Columbia Behavioral Health EQR Report: Clinical Record Review


CLINICAL RECORD REVIEW

In conjunction with the 2013 EDV for GCBH,

Acumentra Health reviewed clinical records at

four outpatient provider agencies to assess mental

healthcare criteria as directed by DBHR. The

clinical record study focused on the degree to

which the RSN’s system of care adhered to the

principles of the Children’s Mental Health

Redesign, including the principles of the interim

settlement of T.R. v. Dreyfus, regarding uniform

screening and assessment of children with serious

emotional disturbances.

The 43 charts reviewed for this activity were the

same as those requested for the EDV. The sample

included consumers age 4–18 served during

October 2011–September 2012, each of whom

had at least three outpatient service encounters

during the review period.

To ensure consistency in reviewing the clinical

records, Acumentra Health followed rigorous

procedures to ensure inter-rater reliability. Before

conducting the review at any RSN, Acumentra

Health trained all reviewers to use a customized

data collection tool and scoring criteria and

guidelines approved by DBHR.

The data collection tool prompted reviewers to

complete a series of questions concerning aspects

of adherence to the T.R. v. Dreyfus principles.

After examining the clinical record (chart) and

progress notes, reviewers recorded responses to

each question in the tool. Using the SAS Proc

Freq function, analysts calculated the distribution

of responses for each question.

Review results

In 71.8% of charts reviewed, an assessment had

been completed within the last year; 23.1% of

assessments had been completed between one and

three years previously, and 5.1% had been

completed three to five years previously.

Within the assessment, 93.0% of records assessed

the child’s living environment and safety needs.

The child’s support systems (95.4%) and activities

outside of the home (97.7%) were assessed in

nearly all records. The assessments usually

considered developmental, learning, or sensory

impairment (90.9%) and language needs (80.0%),

but considered cultural issues less often (63.6%).

The assessment fully justified the diagnosis in

90.7% of records and partially justified the

diagnosis in an additional 4.7% of records.

Nearly all records reviewed contained a treatment

plan that built on strengths to promote resiliency

(92.7%), contained individualized objectives

(95.0%), and included interventions and goals that

were consistent with the assessment (95.1%).

Only 36.8% of treatment plans prescribed team-

based services, and only half (51.2%) had a plan

for case closure.

Nearly two-thirds of treatment plans included

coordination with agencies and others identified

in the assessment, but only 54.5% of progress

notes documented care coordination.

Unconditional treatment was documented in all

progress notes; services were strength-based in

95.4% of records; and progress notes were fully

outcome-based in 80.0% of records, with an

additional 15.0% of records containing progress

notes that were partially outcome-based.

Discussion and recommendations

Assessments: Clinicians at GCBH’s provider

agencies did a good job of documenting the

child’s home environment and systems of support.

The percentage of applicable assessments that

addressed developmental and sensory impairment,

cultural and language issues, and justifying the

diagnosis all fell within the acceptable range.

However, nearly one-third of the assessments were

more than one year old.

Acumentra Health recommends updating

enrollee assessments at least annually to

document changes in the enrollee’s

functioning and life circumstances.



Treatment plans: Most treatment plans reflected

information included in the assessments. The

majority of treatment goals were based on the

children’s strengths. Most records documented

involvement of family members and support

systems in treatment. However, only 64%of the

treatment plans incorporated coordination with

other agencies into the treatment objectives.

GCBH should ensure that agencies

work with their clinicians to coordinate

care with other agencies involved in the

child’s life.

Nearly two-thirds of the treatment plans did not

include a multi-disciplinary team-based approach

to treatment.

GCBH needs to offer guidance to the

agencies to ensure that children’s

treatment includes a team-based

approach when necessary.

Progress notes: All progress notes demonstrated

that the child received unconditional treatment.

The majority of progress notes documented use of

strength-based interventions and activities as

identified in the treatment plans. Team-based

services and coordination with other agencies

were missing from many progress notes. Only

55% of progress notes indicated that services were

team-based and that care was coordinated with

other agencies and systems.

GCBH needs to ensure that services for

children are team-based and that care

coordination occurs. These services

must be documented in the progress

notes to reflect a multi-disciplinary

team-based approach.



Table 9. Results of clinical record review for Greater Columbia Behavioral Health.

Assessment, treatment plan, and progress notes indicate adherence to principles of Children’s Mental Health Redesign

Number of charts reviewed = 43

Assessment % Yes % No

1. Is there a completed assessment within the last year? 71.8 28.2

Assessment includes: % Yes % Partially

1. Living environment and safety needs 93.0

2. Documentation of current living situation

Home (parental) 76.2

Therapeutic foster home 2.4

Other friend/family home 21.4

3. Child’s/family’s natural systems of support 95.4

4. Development, learning, or sensory impairment 90.9

5. Cultural issues that may affect treatment 63.6

6. Language needs taken into consideration 80.0

7. Child/family involvement in activities outside of the home 97.7

8. Justification of diagnosis 90.7 4.7

Treatment plan includes: % Yes % Partially

1. Activities and interventions that build on strengths to promote resiliency 92.7

2. Treatment plan objectives are individualized 95.0

3. Documentation showing family/guardian participation in developing the treatment plan 82.1 2.6

4. Coordination with agencies and collaboration with others identified in assessment 63.6

5. Interventions and goals consistent with issues identified in assessment 95.1

6. Team-based services 36.8

7. Case closure 51.2

Progress notes include: % Yes % Partially

1. Interventions identified in the treatment plan and progress toward meeting stated goals 92.9

2. Unconditional treatment 100.0

3. Documentation that services delivered are strength-based 95.4

4. Progress notes care coordination with agencies and systems 54.5

5. Outcome-based progress notes 80.0 15.0

NOTE: Proportions exclude “not applicable” or missing responses, so the denominator of each item may vary.

[This page deliberately left blank.]

2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol

A-1 Acumentra Health November 2013

APPENDIX A: PIP VALIDATION AND SCORING PROTOCOL

All managed care organizations that serve Medicaid enrollees must conduct two performance

improvement projects (PIPs) each year aimed at improving enrollee health outcomes or processes of

care. The PIPs are validated each year by external quality review to ensure that they are designed,

conducted, and reported according to sound methods.

Acumentra Health’s PIP validation protocol is based on the PIP validation protocol published by the

Centers for Medicare & Medicaid Services (CMS). The most recent version of the CMS protocol,

published in September 2012, added a requirement related to cultural competency. For 2013, Acumentra

Health added a criterion to the DBHR PIP Review Tool to address this requirement:

6.4 Discuss how the intervention services and materials are culturally and linguistically

appropriate.

Other changes in the 2012 CMS protocol reflect a new emphasis on certain aspects of the study design.

Acumentra Health will modify the PIP Review Tool for 2014 to conform to the new areas of emphasis

in the CMS protocol.

The 10 validation standards adapted by Acumentra Health from the CMS protocol define critical

elements in a PIP study design. Specific criteria for each standard are listed on the following pages.

Standard 1. Study Topic

Standard 2. Study Question

Standard 3. Study Indicator

Standard 4. Denominator (Study Population) Data Collection

Standard 5. Numerator (What Is Being Measured) Data Collection and Analysis Plan

Standard 6. Study Intervention

Standard 7. Study Results

Standard 8. Interpretation of Study Results

Standard 9. Study Modifications After the First Remeasurement

Standard 10. Overall Study Results



Standard 1: Study Topic

To meet Standard 1, the RSN needs to establish the importance of the study topic in general and present

local data to demonstrate that the topic applies to a large or high-risk portion of the Medicaid

population; and demonstrate that a systematic selection and prioritization process was used in choosing

the topic.

Please explain:

1.1 The importance of the study topic in general.

1.2 How the study topic is relevant to your local Medicaid population.

1.3 How you identified the study topic (e.g., quality committee, focus group, grievances, QAPI

activities, other sources).

1.4 Why you prioritized this topic, including considerations of quality (e.g., high risk, prevalent

issue) and feasibility (e.g., data and resource availability).

1.5 How the study topic relates to enrollee outcomes, satisfaction, or quality of care.

Standard 2: Study Question

To meet Standard 2, the RSN needs to present a study question that provides a clear framework for data

collection, analysis, and interpretation. The study question should refer to the proposed intervention, a

study population (denominator), a measure (numerator), a metric (e.g., average, percentage), and a

direction of desired change.

2.1 Please state your study question. A complete study question includes an intervention, a study

population (denominator), what you are measuring (numerator), a metric (percent or average),

and a desired direction of change (increase or decrease). If you have more than one study

indicator, you should present a separate study question for each study indicator.

Standard 3: Study Indicator

To meet Standard 3, the RSN needs to define the measure (numerator) and study population

(denominator); define key terms; and discuss the basis for adopting the indicator as a valid proxy for

enrollee outcomes, satisfaction, or quality of care.

Please define the following elements for each study indicator:

3.1 The denominator (study population), and continuous enrollment criteria if applicable.

3.2 The numerator (what is being measured), including the event or enrollee characteristics that

qualify for the numerator.

3.3 All relevant terms, CPT codes, diagnosis codes, etc., associated with the study indicator.

3.4 Describe why you selected the study indicator. Your description should include a discussion of:

a. The validity of the study indicator (i.e., HEDIS, commonly accepted measures, research

literature, etc.).

b. How the indicator measures enrollee outcomes, satisfaction, or quality of care either directly

or indirectly through a process which is closely related to enrollee outcomes or satisfaction.



Standard 4: Denominator (Study Population) Data Collection

To meet Standard 4, the RSN needs to list all inclusion and exclusion criteria for the study population;

document all data sources, including fields, codes, and calculations; and describe data validation

procedures. If a sample is selected, the RSN needs to describe the sampling methods.

Describe your data sources.

4.1 List the inclusion criteria for the denominator (study population) and name each data element

and its source, table, field, calculation (if applicable), and relevant codes.

4.2 List all exclusion criteria for the denominator (study population) and name each data element and

its source, table, field, calculation (if applicable), and relevant codes. You do not need to list the

inverse of the inclusion criteria as exclusions.

4.3 Describe data validation procedures for each data element.

4.4 If you used a sample, describe the sampling methodology and a justification for the sample size.

Standard 5: Numerator (What Is Being Measured) Data Collection and Analysis Plan

To meet Standard 5, the RSN needs to list all inclusion and exclusion criteria for the numerator (what is

being measured); document all data sources, including fields, codes, and calculations; describe data

validation procedures; and present a clear data analysis plan, including time frames for the

measurement and intervention periods, and an appropriate statistical test to measure differences

between the baseline and remeasurement periods.

Describe your data sources.

5.1 List the study inclusion criteria for the numerator (what is being measured) and name each data

element and its source, table, field, calculation (if applicable), and relevant codes.

5.2 List all exclusion criteria for the numerator (what is being measured) and name each data

element and its source, table, field, calculation (if applicable), and relevant codes. You do not

need to list the inverse of the inclusion criteria as exclusions.

5.3 Describe data validation procedures for each data element.

5.4 Document clear study measurement periods. The baseline period should end before the start date

of the intervention. The first remeasurement period should not begin before the start date of the

intervention. The intervention and remeasurement periods may run concurrently.

5.5 Document a data analysis plan that includes an appropriate statistical test, rationale for selecting

the test, and a probability level. If you have more than one study indicator, you should document

a separate data analysis plan for each indicator.



Standard 6: Study Intervention

To meet Standard 6, the RSN needs to select an improvement strategy that will affect a wide range of

enrollees or a high-risk enrollee population, and that is reasonably expected to result in measurable

improvement. The RSN needs to discuss the basis for adopting the intervention; document the

implementation of the intervention, including dates and locations of principal activities; discuss cultural

competence; and track how effectively the intervention was implemented.

6.1 Describe the intervention strategy. Once intervention activities begin, please provide updated

details, including dates and locations.

6.2 Describe why you selected this particular intervention; for example, because it is based on

barriers identified in your system or because it is an evidence-based practice. It should be clear

how the intervention strategy is expected to improve the study indicators.

6.3 Describe how you will track the implementation of the intervention (i.e., how you will know

whether all aspects of the intervention were implemented successfully). If the intervention has

already been implemented, report on the results of your tracking.

6.4 Discuss how the intervention services and materials are culturally and linguistically appropriate.

Standard 7: Study Results

To meet Standard 7, the RSN needs to present results according to the data analysis plan, including the

study indicator, the original data used to compute the indicator, and a statistical test to measure

differences between the baseline and remeasurement periods; and discuss how the intervention

influenced the results.

7.1 Present raw data for the numerator (what you are measuring) and denominator (study population)

as well as the calculated study indicator for the baseline and first remeasurement periods.

7.2 Present the results of your statistical analysis comparing baseline data to the first remeasurement

data. Report the probability level to determine whether or not there is a statistically significant

difference.

7.3 Discuss how the intervention influenced the study results.

Standard 8: Interpretation of Study Results

To meet Standard 8, the RSN needs to assess whether any reported improvement is “real” by

documenting that baseline and remeasurement data were collected using the same methods and are

comparable; discuss the statistical and clinical significance of the study results; address barriers to

improvement and lessons learned during the PIP process; and identify confounding factors that may

have affected the results.

Discuss the following:

8.1 Whether the PIP resulted in real statistical and clinical improvement.

8.2 Any barriers to improvement or lessons learned during the PIP process.

8.3 Whether there were any changes in methodology or inconsistencies in measurement periods and,

if so, whether measurement periods are comparable.

8.4 Any confounding factors that may have affected the PIP results.



Standard 9: Study Modifications After the First Remeasurement

To meet Standard 9, the RSN needs to document modifications to the intervention, or added

interventions, planned or implemented after the first remeasurement period; and discuss changes in

other aspects of the PIP based on lessons learned from data analysis or barrier analysis.

9.1 Discuss how you addressed the identified barriers and describe any other modifications you

made to the PIP after the first remeasurement period.

Standard 10: Overall Study Results

To meet Standard 10, the RSN needs to report complete study results for two or more measurement

periods, including the study indicator, original data used to compute the indicator, and a statistical test

of group differences; and interpret the statistical and clinical significance of the overall results, discuss

lessons learned, and determine if goals were met and sustained improvement was achieved.

10.1 Present raw data for the numerator (what you are measuring) and denominator (study

population), and the calculated study indicator for the baseline and the second remeasurement.

10.2 Present the results of a statistical analysis comparing baseline data to the second remeasurement

data. Report the probability level to determine whether or not there is a statistically significant

difference.

10.3 Interpret whether the PIP resulted in sustained statistical and clinical improvement over multiple


10.4 Draw a conclusion about whether the PIP was successful overall. Discuss lessons learned during

the PIP process, whether you met your goals for this PIP overall, and the factors that contributed

to whether the PIP achieved sustained improvement.



PIP scoring

Acumentra Health assigns a score to each PIP standard to measure compliance with federal standards.

Each standard has a potential score of 100 points, as shown in Table A-1.

Table A-1. Compliance rating for PIP standards by point range.

Rating Definition Points

Fully met Meets or exceeds the essential criteria 100

Substantially met Meets essential criteria, has minor deficiencies 75–99

Partially met Meets criteria with deficiencies in some areas 50–74

Minimally met Marginally meets criteria 25–49

Not met Does not meet essential criteria 0–24

The scores for each standard are weighted and combined to determine the overall PIP score. The

maximum overall score is 90 points for Standards 1–8, and 100 points for Standards 1–10, as shown in

Table A-2.

Table A-2. Weighting of points on PIP standards in the overall PIP score.

Standard Scoring weight

1 Study Topic 20%

2 Study Question 10%

3 Study Indicator 10%

4 Denominator (Study Population) Data Collection 10%

5 Numerator (What Is Being Measured) Data Collection and Analysis Plan 10%

6 Study Intervention 10%

7 Study Results 10%

8 Interpretation of Study Results 10%

Demonstrable Improvement Score 90%

9 Study Modifications After the First Remeasurement 5%

10 Overall Study Results 5%

Sustained Improvement Score 10%

Overall PIP Score 100%



The overall PIP score corresponds to a compliance rating that ranges from Fully Met to Not Met.

Table A-3 shows the compliance ratings and associated scoring ranges for PIPs graded on the 90-point

and the 100-point scale.

Table A-3. Compliance rating for PIPs by overall score.

Compliance rating Description

100-point scale

90-point scale

Fully met Meets or exceeds all requirements 80–100 72–90

Substantially met Meets essential requirements, has minor deficiencies

60–79 54–71

Partially met Meets essential requirements in most, but not all areas

40–59 36–53

Minimally met Marginally meets requirements 20–39 18–35

Not met Does not meet essential requirements 0–19 0–17

Table A-4 shows an example scoring calculation for a PIP on Standards 1–8 for demonstrable

improvement, and on Standards 1–10 for sustained improvement.

Table A-4. Scoring worksheet example.

Standard Compliance rating Assigned

points Weight Overall score

1 Fully met 100 20% 20.00

2 Fully met 100 10% 10.00

3 Partially met 50 10% 5.00


5 Fully met 100 10% 10.00

6 Minimally met 25 10% 2.50



Overall score 1–8 Substantially met 62.50

9 Substantially met 75 5% 3.75


Overall score 1–10 Substantially met 68.75

2013 Greater Columbia Behavioral Health EQR Report: Appendix B: ISCA Methodology

B-1 Acumentra Health November 2013

APPENDIX B: ISCA METHODOLOGY

The Information Systems Capabilities Assessment (ISCA) examines each RSN’s information systems

and data processing/reporting procedures to determine the extent to which they support the production of

valid and reliable state performance measures and the capacity to manage care of RSN enrollees.

The ISCA protocol for the Washington RSNs is adapted from the protocol published by the Centers for

Medicare & Medicaid Services (CMS). The review process consists of four phases:

Phase 1: Collect standard information about RSN information systems. The RSN completes the

ISCA data collection tool (ISCA-T) provided by Acumentra Health before the onsite review.

Acumentra Health also asks the RSN to submit other relevant documents at this time.

Phase 2: Acumentra Health reviews the completed ISCA-T and accompanying documents. Where

an answer seems incomplete or indicates an inadequate process, Acumentra Health marks that

section for follow-up.

Phase 3: Data center security walkthrough and in-depth interviews with knowledgeable RSN staff.

Provider agency interviews, also performed at this time, ask about each agency’s information

systems, encounter/claims processing, and handling of enrollment data.

Phase 4: Post-onsite analysis of findings about the RSN’s information systems and the implications

of the findings regarding:

1. completeness and accuracy of claims and encounter data collected and submitted to DBHR

2. the RSN’s capacity to conduct quality assurance/performance improvement initiatives

3. the RSN’s capacity to oversee and manage the delivery of health care to its enrollees

The following pages discuss the specific criteria for assessing compliance with each review standard.

Section 1: Data Processing Procedures and Personnel

Section 1A: Information Systems

This section provides a detailed review of the RSN’s systems development life cycle (SDLC) and

supporting environments, including database management systems and/or billing software,

programming languages, and training for programmers.

A data storage and processing system that facilitates valid and reliable performance measurement would

have the following characteristics:

flexible data structures

no degradation of processing with increased data volume

adequate programming staff

reasonable processing and coding time

ease of interoperability with other database systems

data security via user authentication and permission levels

data locking capability

proactive response to changes in encounter and enrollment criteria

adherence to the federally required format for electronic submission of encounter data



To ensure accurate and complete performance measure calculation, best practices in computer

programming include:

good documentation

clear, continuous communication between the client and the programmers on client information

needs (e.g., analysis needs, reports)

a quality assurance process

version control

continuous professional development of programming staff

Section 1B: Staffing

This section assesses physical access by the RSN’s staff to IT assets, as well as specific training

requirements for claims processing staff.

Best practices for sustaining quality in processing encounter data include

adequately trained staff for processing and tracking errors in encounter data submission

a comprehensive, documented formal training process for new hires and experienced

professionals

refresher courses for staff when updates occur and when new systems are implemented

established and monitored productivity goals

low staff turnover

Section 1C: Hardware Systems

This section assesses the RSN’s network infrastructure and hardware systems.

Best practices for sustaining quality hardware systems include

infrastructural support that includes maintenance and timely replacement of computer equipment

and software, disaster recovery procedures, adequate training of support staff, and a secure

computing environment

redundancy or duplication of critical components of a hardware system with the intention of

increasing reliability of the system, usually in the case of a backup or fail-safe

Section 1D: Security

This section assesses the RSN’s information systems for integrity and the ability to prevent data loss and

corruption. A security walkthrough of the computer area and/or data center assesses the possibility of a

breach in security measures.

Best practices for securing data are summarized below.

A well-run security management program includes IT governance, risk assessment, policy

development, policy dissemination, and monitoring. Each of these activities should flow into the

next to ensure that policies remain current and that important risks are addressed.

Computer systems and terminals should be protected from unauthorized access through use of a

password system and security screens. Passwords should be changed frequently and reset

whenever an employee terminates.



Paper-based claims and encounters should be in locked storage facilities when not in use.

Data transferred between systems/locations should be encrypted.

A comprehensive backup plan includes scheduling, rotation, verification, retention, and storage

of backups to provide additional security in the event of a system crash or compromised integrity

of the data. Managers responsible for processing claims and encounter data must be

knowledgeable of their backup schedules and of retention of backups to ensure data integrity.

To ensure integrity, backups should be verified periodically by performing a “restore” and

comparing the results. Ideally, annual backups would be kept for seven years or more in an

offsite climate-controlled facility.

Databases and database updates should include transaction management, commits, and rollbacks.

Transaction management is useful when making multiple changes in the database to ensure that

all changes work without errors before finalizing the changes. A database commit is a command

for committing a permanent change or update to the database. A rollback is a method for

tracking changes before they have been physically committed to disk. This prevents corruption

of the database during a sudden crash or some other unintentional intervention.

Formal controls in the form of batch control sheets or assignment of a batch control number

should be used to ensure a full accounting of all claims received.

Section 11.2 of DBHR’s 2011–2013 RSN contract presents requirements related to Business Continuity

and Disaster Recovery (BC/DR). The contractor must certify annually that a BC/DR plan is in place for

both the contractor and subcontractors. The certification must indicate that the plans are up to date and

that the system and data backup and recovery procedures have been tested. The plan must address:

a mission or scope statement

an appointed IS disaster recovery staff

provisions for backup of key personnel, identified emergency procedures, visibly listed

emergency telephone numbers

procedures for allowing effective communication with hardware and software vendors

confirmation of updated system and operations documentation, process for frequent backup of

systems and data

offsite storage of system and data backups, ability to recover data and systems from backup files

designated recovery options that may include use of a hot or cold site

evidence that disaster recovery tests or drills have been performed

Exhibit C of the 2011–2013 RSN contract presents detailed requirements for data security, including:

1. data protection during electronic transport, including via email and the public Internet

2. safeguarding access to data stored on hard media (hard disk drives, network server disks, and

optical discs), on paper, or on portable devices or media, and access to data used interactively

over the State Governmental Network

3. segregation of DSHS data from non-DSHS data to ensure that all DSHS data can be identified

for return or destruction, and to aid in determining whether DSHS data has or may have been

compromised in the event of a security breach

4. data disposition (return to DSHS or destruction) when the contracted work has been completed

or when data no longer needed

5. notification of DSHS in the event of compromise or potential compromise of DSHS shared data

6. sharing of DSHS data with subcontractors



Section 2: Data Acquisition Capabilities

Section 2A: Administrative Data

This section provides a detailed review of the RSN’s submission of accurate information, process for

describing differences when verifying accuracy of submitted claims, and data assessment and retention.

To ensure the validity and timeliness of the encounter and claims data used in calculating performance

measures, it is important to have documented standards, a formal quality assurance of input data sources

and transactional systems, and readily available historical data.

Best practices include:

automated edit and validity checks of procedure and diagnosis code fields, timely filing,

eligibility verification, authorization, referral management, and a process to remove duplicate

claims and encounters

a documented formal procedure for rectifying encounter data submitted with one or more

required fields missing, incomplete, or invalid. Ideally, the data processor would not alter the

data until receiving written notification via a paper claim or from the provider.

periodic audits of randomly selected records conducted internally and externally by an outside

vendor to ensure data integrity and validity. Audits are critical after major system upgrades or

code changes.

multiple diagnosis codes and procedure codes for each encounter record, distinguishing clearly

between primary and secondary diagnoses

efficient data transfer (frequent batch processing) to minimize processing lags that can affect

data completeness

Section 2B: Enrollment Systems

This section assesses the RSN’s Medicaid enrollment systems pertaining to enrollment and

disenrollment processes, tracking claims and encounter data, Medicaid enrollment data updates,

Medicaid enrollment code, and data verification.

Timely and accurate eligibility data are paramount in providing high-quality care and for monitoring

services reported in utilization reports.

Best practices are summarized below.

Access to up-to-date eligibility data should be easy and fast.

Enrollment data should be updated daily or in real time.

The enrollment system should be capable of tracking an enrollee’s entire history within the RSN,

further enhancing the accuracy of the data.

Section 2C: Vendor Data Integrity

This section assesses how the RSN integrates vendor data with administrative data for completeness of

data and quality of data.

An ideal vendor data integration system includes:

converting data, including code sets, for compatibility with the state’s data systems



receiving only member-level data, as opposed to aggregate data

incorporating other data (e.g., dental care, primary care manager, history of care) to provide a

more complete picture of a member’s care

ensuring consistency in the data for required fields, including multiple diagnosis and procedures

codes

Section 2D: Provider Data

This section assesses how the RSN maintains its provider directory, as well as the RSN’s fee schedules

and contractual payment updates.

An RSN designs its provider compensation structure to balance contractual expectations, the needs of

enrolled populations, and capitation rates set by the state. To set appropriate capitation rates, the state

relies on accurate and timely encounter data.

A good payment structure is critical to ensure reasonable and timely compensation, which encourages an

accessible, qualified community network of providers to continue to provide service to Medicaid

enrollees.

An ideal provider profile directory, which allows enrollees and staff to make informed choices, would

list all available providers, including their gender, credentials and specialties, languages spoken, whether

they use sign language, whether they offer interpretive services, and whether the office is ADA-

certified.

Section 2E: Meaningful Use of Electronic Health Records

This section requests information on how the RSN and its contracted providers use electronic health

records (EHRs), including:

any planning and/or development efforts the RSN has taken toward adopting and using a

certified EHR system

number of providers in the RSN network currently using EHRs

if providers are using EHRs, whether the RSN has sought to determine whether the technology

has been certified by the appropriate federal body

any training, education, or outreach the RSN has delivered to network providers on the

meaningful use of certified EHR technology

whether the RSN uses data from EHRs as part of its quality improvement program (e.g., to

improve the quality of services delivered or to develop PIPs)

strategies or policies the RSN has developed to encourage the adoption of EHR by providers that

are not eligible for the Medicaid Incentive Program

2013 Greater Columbia Behavioral Health EQR Report: Appendix C: Encounter Data Validation Procedures

C-1 Acumentra Health November 2013

APPENDIX C: ENCOUNTER DATA VALIDATION PROCEDURES

In validating RSN encounter data, Acumentra Health follows the steps outlined below, based on the

CMS protocol, Validating Encounter Data.

1. Review the state’s requirements for collecting, processing, and submitting encounter data, based

on specifications in the RSN contract, the state’s data dictionary, and other information furnished

by the state.

2. Review results of the previous EDV study to identify follow-up needs.

3. Review the capability of each RSN’s information system to capture accurate and complete

encounter data, drawing on findings of the ISCA review and interviews with RSN personnel.

4. Analyze electronic encounter data to establish the magnitude of missing data, types of potentially

missing data, overall data quality issues, and problems with how the RSN compiles and submits

encounters to the state. Subtasks include:

Apply general edit and consistency checks, such as verifying that critical fields contain

values that are consistent across fields.

Inspect data fields for general validity, including a review of each data element and of the

volume of data by type or place of service.

Using standard statistical procedures, analyze data to obtain a validity overview of the RSN’s

encounter data. This step involves analyzing and interpreting the data in submitted fields, the

volume and consistency of encounter data, and utilization rates, both overall and by specific

diagnosis, procedure, service, and provider types.

Compare the RSN’s encounter data with state standards and/or benchmarks.

5. If necessary, review clinical records to confirm findings of the above analysis.

Acumentra Health reviewed each RSN’s internal EDV activities in 2013, but unlike in 2012, did not

recreate the EDV calculations. Based solely on the information provided in the RSN’s EDV report,

Acumentra Health assessed whether the RSN’s EDV tool, sampling procedures, EDV process, and

subsequent results were adequate for assessing the accuracy and completeness of the EDV data.

Acumentra Health reviewed a sample of enrollee charts to ensure that the information in the encounter

data matched the information in the charts. The following data elements were scored:

first name

last name

date of birth

ethnicity

language

provider type

minutes of service

service location

service date

procedure code

whether the service code matched the chart note

2013 Greater Columbia Behavioral Health EQR Report: Appendix C: Encounter Data Validation Procedures

C-2 Acumentra Health November 2013

Scoring options included:

Match: cases where there is an exact match of all the minimum data elements for each randomly

selected sample between the subcontractor’s encounters and those in the clinical record

No Match: cases where the subcontractor’s encounters do not match the clinical records.

o Erroneous: Encounters occurred and are presented in the clinical record but contain incorrect

data or omit any of the minimum data elements.

o Unsubstantiated (not in the medical record): Encounters submitted by the subcontractor

either cannot be verified in the clinical record or are duplicated.

o Missing (not in the encounter record): Clinical record contains evidence of a service but the

service is not represented by the encounter record.

Also in 2013, Acumentra Health reviewed all sample enrollee charts for evidence of adherence to the

“Golden Thread” of therapy, evaluating whether the assessment substantiates the diagnosis, whether the

treatment plan is consistent with the diagnosis, and whether progress notes address the treatment plan.

The following Golden Thread data elements were scored:

Does the assessment substantiate the Category A diagnosis?

Does the assessment substantiate the Category B diagnosis?

Does the treatment plan include interventions and goals consistent with issues identified in the

assessment?

Do the progress notes address interventions identified in the treatment plan and the individual’s

progress toward meeting stated goals?

Are treatment plan objectives individualized?

Documents

Greater Columbia Behavioral Health - GCBH · Summary results of Greater Columbia Behavioral Health PIP validation ... Validation scores by standard for clinical PIP, Lowered PRISM