1

2nd International Conference on Government Performance Management and Leadership

Draft Conference Paper

October 1st-2nd, 2011

Title: Intergovernmental barriers and the development of state performance management

systems: Results from the federal National Outcome Measures project

Author: Carl Foreman, Doctoral Student Portland State University

Abstract

Federal and state agencies must cooperate to provide mental health services, which

creates a complex implementation environment. This environment can cause significant

challenges for measuring agency performance and accountability. Complex

implementation chains are a functional component of federal programs. This requires

states to perform a central role in the reporting and ensuring of accountability. Viewed

from a state perspective, federal programs create a dilemma for reporting agencies. In

particular, federal block grants have various levels of funding that are not always aligned

with the reporting requirements. Misalignment between funding and reporting can cause a

2

disproportionate impact on agency resources. Federal programs that require significant

reporting through administrative data sets are the most easily accessible input for the

development of state-level performance management systems. There are limitations in the

use of administrative data in estimating program impact although the data has been proven

useful in identifying policy levers (Heinrich, 2002). This study examines the usefulness of

federally collected data for the development of state performance management systems.

Despite efforts to collect standardized national data at the federal level, states were found

to not sufficiently report on evidence-based practice measures to provide useful, in-depth

information for performance management systems. However, the results do provide

information regarding the lack of monitoring of evidence-based practices. This information

is useful to states in their decision to allocate resources when conflicting state and federal

goals are present.

Introduction In Int

SAMSHA began tracking ten National Outcomes Measures in FY 2004 with the goal

of improving state performance management systems (Manderscheid, 2006). These

measures were included in the state’s federal block grant reporting requirements. There

are two federal block grants associated with the provision of behavioral health services.

The Community Mental Health Block Grant (CMHBG) is administered through the Center

for Mental Health Services (CMHS), under the Substance Abuse and Mental Health Services

Administration (SAMSHA, 2011a). The second behavioral health related block grant is the

Substance Abuse Prevention and Treatment (SAPT) grant which was administered by the

3

Center for Substance Abuse Treatment (CSAT) and Center for Substance Abuse Prevention

(CSAP) (SAMSHA, 2011a). Beginning in FY 2012, the two grants will be combined into a

uniform block grant application (SAMHSA, 2011). In 2009, the National Association of

Mental Health Program Directors National Research Institute (NRI) redeployed the state

level National Outcomes Measures data collection in nine states to increase the quality of

captured data (National Association of State Mental Health Program Directors Research

Institute, Inc., 2009). The National Outcome Measures were planned to be used in

conjunction with a decision support system that was never fully implemented

(Manderscheid, 2005). Evidence-based practices represent one domain in the National

Outcome Measures; evidence-based practices have also been a policy consideration of the

states (Rieckmann, et al., 2008). The State of Oregon began a titrated implementation of a

state legislative mandate requiring a 25% increase in purchasing of evidence-based

practices by biennium (Oregon Revised Statute 182.525). The state of Oregon had the

similar goal of increasing the use of evidence-based practices as SAMSHA but chose a

different implementation plan. This plan mandated the purchase of evidence-based

practices while the federal agency did not use a mandate and selected ten evidence-based

practices that were chosen through a national consensus process.

The results point to the need for further efforts to increase data quality in order to

provide sufficient data for state and federal decision-making. This exploratory analysis

focused on the quality of the national outcome data in providing useful performance

information to the states. The results point to a larger issue in the sustainability of

performance management systems such as increasing the usefulness of performance data

across intergovernmental agencies.

4

Performance Management Systems: Definitions and Requirements

Bird et al. (2005) identify three functions of performance monitoring in the public

sector: determining the policy impact, determining the level of agency performance, and

stewardship of public funds. Pidd (2007) adds the support of central control and symbolic

action as the reasoning for the pervasive use of performance monitoring in the public

sector. Specific to health care, Commission on Behavioral and Social Sciences and Education

(CBASSE) describes good performance measures as aimed at specific objectives and

supported by adequate data. They should also be sufficiently valid, reliable, and responsive

to changes in the environment (Perrin & Koshel, 1997). Third-party governance adds

complexities into the use of performance measures to ensure accountability or

performance (Posner, 2002). In third-party governance, performance occurs with a

contracted entity and does not necessarily reflect the performance of the state agency

monitoring the contract. In the practical world of system delivery in a third-party

governance structure, completely valid and reliable data is not necessarily available.

Therefore, performance management requires providing information for administrative

decision-making from sufficiently accurate data sources from within and outside the

agency.

Performance Measures

Performance Measures need to be able to meet the internal management needs

across multiple purposes (Behn, 2003). Dubnick and Frederickson (2009) developed six

categories of accountability used in the Department of Human Services. Dubnick and

Fredrickson characterize federal performance measures as relating management

5

techniques to political processes (2009). Additionally, Dubnick and Fredrickson (2009)

state that there is a lack of evidence that performance measures truly impact issues of

equity in public law. Issues of equity and individual rights are a fundamental component in

the provision of mental health services. There are definite limits to the expectations for

performance measures and management. While performance measures and management

provide information regarding certain aspects of performance, they do not provide all of

the information regarding the activities of the agency. These limitations are especially

pronounced in areas where there are long implementation chains, and the state agency is

accountable to multiple stakeholders. Wright views the relationship of federal, state, and

local governments through an overlapping authority model. In this model, each layer of

government is interdependent, has overlapping relationships and exerts authority through

bargaining with the other intergovernmental layers (Wright, 1988). There are some

challenges in the interaction of intergovernmental layers. For example, federal grants

provide the federal government with a voice in state decision-making; however, this role

causes a diffusion of responsibility among federal and state representatives with their

respective constituents (Derthick, 1970, p.214).

Third-Party Implications

Performance measures are directed to efficiencies and effectiveness, but they do

not capture the total tasks of the agency or the interaction of agencies with third-party

contractors. Mechanisms for addressing equity have similar challenges in capturing

adequate data in order to make informed decisions. For example, a grievance process for

state contracted services can be initiated at several different access points. For mental

6

health services, which provide services to a vulnerable population, there are several outlets

for informal grievances. Included are: the provider, the managed care organization,

advocacy council, the managed care ombudsman and various other agencies depending on

the exact nature of the complaint. For Medicaid clients there is also an annual survey

required for customer satisfaction. These multiple avenues for the grievance process

provide for the opportunity of quick informal resolution of the grievance. However,

multiple grievance referral sources create challenges for quantification and may not

provide an accurate portrayal of the exact nature of the grievance. Customer surveys are

amenable to performance measurement, but capture a sample of the population and are

susceptible to response bias. The performance management aspect of the grievance

procedure that is most relevant to the state agency is the presence of timely controls that

can address issues that are addressed in grievances. For issues such as access to services,

the grievance procedure is a symptom of a systemic problem which, when quantified, may

not satisfactorily capture the nature of the grievance to suitably provide the necessary

action to address the systemic flaw. Quantifying the number of grievances provides a data

element for managerial decisions, but it is not solely adequate to make administrative

decisions. The difficulty is determining the sufficient data elements at the system level

required to make useful decisions.

Framework for Performance Measures

Performance measures have a vital role in quality assessment in behavioral health.

Performance measures are included with activities such as audits and accreditation which

provide external accountability and grievance systems. Performance measures accomplish

7

multiple goals measures focused on internal improvement and monitoring external and

internal processes (Edmunds, M. et al.,1997, p.97). Each of the performance measures in

this complex framework interaction to varying degrees with agencies in the

intergovernmental system. Performance measures are tracked differently at these levels

which can cause redundancies in reporting. While this complex framework provides

sensitivity to the reporting needs of the various reporting agencies, the collectors or

monitors of the information normally reside at the state agency. While survey information

can be populated automatically if used from an external data source, performance data

especially the tracking of innovative practices such as evidence-based practices requires

data collection at the provider level. In this framework, the state agency is in the

predominate role of verifying data quality. The ability of the state agency to assure the

quality of the data is limited by the capacity of the state agency to track data. The ability of

the state to track data quality is limited by the capacity of the providers to collect data.

Challenges to performance measurement

There are several challenges to the implementation of performance measures in the

public sector. One of the fundamental challenges is that performance measures can be

used for multiple purposes that may not be accurately reflected in the measure.

Performance measures are used for accountability and performance, but their ability to

meet both of these purposes is difficult. For example, efforts to use performance measures

to increase performance through accountability can have deleterious effects on

performance (Halachmi, 2002). Increases in accountability do not necessarily translate into

8

rises in agency performance. Dedicating resources to become more accountable takes

resources away from performance tasks. Performance measures are used to provide

information at several levels of government for accountability and performance purposes

which is further complicated by their use as measures of accountability and performance.

Several of these performance measures are mandated by the funding agencies. Mandated

performance measures can cause greater challenges than recommended measures

(Halachmi, 2002). Mandated performance measures provide information relevant to the

recipient agency but are not sensitivity to implementation and factors and constraints at

the state level. There are multiple roles for measures in the public sector.

Internal and External Focused Measures

One level of distinction between performance measures is internal focused

measures and measures of external focus. In public health, external measures provide view

of community health which relate to internal performance but involve contextual factors

and areas that cross multiple agency agendas. Community level measures provide

information regarding the contextual environment that state agencies operate. While

external measures track the overall health of the community they are not necessarily

factors of the agency performance or accountability. State agencies can make decisions on

resource allocation based on information obtained from performance measures but if the

performance measures track domains beyond the reach of the agency then there is a

disincentive to incorporate decision-making based on performance management and

instead focus on other administrative or political decision-making strategies. Objective

information regarding broad community metrics does not provide the agency with a

9

distinct policy lever from which to improve performance or accountability. In fact broad

based external measures may serve to disengage in the use of performance measures as an

element for decision-making and instead serve as a political justification for more funding.

Performance measures are also functions of political and contextual variations

present at each governance level. Federal performance measures provide information for

the executive branch and congress while state level performance measures are directed

toward state legislatures. State agencies and the providers that they monitor have to collect

information to meet the reporting requirements for both federal and state legislative

bodies. State administrative agencies have limited resources and bounded rationality

regarding management decisions. In order to meet the demands from these multiple

stakeholders and internal management decisions, choices have to be made on how to

approach implementation monitoring. Political decisions at the state and federal level

impact what a state agency can focus on. Even when there is agreement in the goals

between the state and federal levels, there may be disagreement on what elements of

performance to track. When there are distinctions between the federal and state measures,

the state must make resource allocation decisions based on the relative importance of the

performance measure.

Structural Barriers

There are several structural factors that impact public health performance. Mays et.

al. (2006) found that public health performance had varying rates of impact based on the

size, financial resources, and organizational resources of the local public health system.

When using federal performance measures several contextual factors along multiple

10

governance layers can impact the performance measures. There is also uncertainty

regarding the exact mechanism in which structural factors impact public health outcomes

(Mays et al., 2009). One of the challenges is separating changes in program eligibility and

programs that occur through legislative decisions and those related to agency discretion.

Statutory barriers also impact the ability of an agency to impact performance measures or

may spur the development of performance measures to track legislative compliance. There

are many political and contextual reasons beyond management decision-making that

influence performance measure selection. There are methods to address the contextual

factors associated with agency performance. Adjusted performance measures which use

contextual factors that impact agency performance (Rubenstein, 2003). While adjusted

performance measures are sensitive to contextual factors they are unable to account for the

political influences at each level of the complex implementation cycle

State and Federal goal alignment

One for the challenges presented to state mental health agencies is goal ambiguity at

the federal level. Ho (2007) describes the differences in the definition of performance

between the branches of government with regards to the Substance Abuse and Mental

Health Agency. The executive branch viewed performance through the lens of cost-

effectiveness whereas congress viewed performance as a measure of meeting stakeholder

needs (Ho, 2008, p.329). A variety of definitions of performance is also present at the state

level. There have been numerous efforts to monitor state and federal performance, and

some of these efforts are not implemented with incentives. In a model without incentives,

accountability professionals can serve as the instrument of performance monitoring. The

11

task of the accountability agency is to objectively determine if the implementation of the

performance measure will impact performance or accountability. However, there are

challenges in depending on external accountability professionals to determine agency

performance. One of the challenges is the differences in values between accountability

professionals and agency staff (Power, 1997). From a practical programmatic standpoint,

without a performance linked incentive, there is little reason to assume that a performance

report will result in program improvement. Given the number of mandated performance

measures in which agencies must track, measures without incentives by definition become

less of an agency concern. However, there are instances when the agency actions are

aligned with the federal performance measures. Given the resources required at the state

level to ensure that providers track data and the performance of data integrity on the

collected data it is instances when incentives are used that the state and federal agency are

the most likely to be aligned.

Innovation and performance

One of the uses of performance measures is to track the implementation of a service

innovation. Tracking an innovation has additional challenges due to the fact that innovation

is also moderated by internal agency factors and interagency networks. In their analysis of

factors contributing to the adoption of matrix management at organizations Burns and

Wholey (1993) determined that task diversity, socio-metric location, dissemination of

information, and the cumulative force of adoption influenced the adoption of innovative

management practices. Federal performance measures are able to have a limited impact

on the dissemination of information and the cumulative force of adoption. The limited

12

impact of performance measures on the organizational factors that influence change in

combination with the lack of a performance incentive serves to mitigate the impact of the

performance measure. Additional questions regarding the configuration of the

performance management system are the number of measures and the alignment of

measures at various governance levels. Public accountability takes many forms which

require different administrative actions, one of which is performance monitoring and

management. Dubnick & Frederickson, (2009) note that federal performance measures

represent accountability to the executive branch and there but there is limited evidence on

impact on issues of equity and justice.

Agency Motivation

State mental health agencies receive signals from national reporting systems such as

the Uniform Reporting System regarding the value of the types of performance data.

However, without incentives or administrative support, the measures tracked in the URS

data are of limited use to the state and agency decision-making. In an area such as

evidence-based practices which is an innovative practice, there are difficulties associated

with its measurement. The tracking of the use of innovative measures requires changes to

state administrative systems which maybe legacy systems. Similarly, providers may not

have a method of tracking these new measures. Tracking these measures can require the

agency to invest in supplemental datasets or other methods to obtain the performance

information. These supplemental systems may have limited protections regarding the

validity and reliability of the submitted data. The performance measures may be tracking

13

the ability of an administrative system to track the innovative measures rather than the

diffusion of innovative practices.

Data Quality

Data quality is a concept that is constructed of multiple components and lacks a

standardized definition (Pipino et al., 2002; Mitchell and Brown, 2002). The nature of

performance measures make maintaining data quality a challenge. Radin (2000) highlights

the challenges of finding valid data that meets the needs of the various stakeholders

representing multiple levels of governance. In addition to the federal state interface, federal

block grants also interact with local governments. While information may report up to the

superordinate agency, information does not always flow downward to the subordinate

agency. This lack of integration of data reduces the probability that the data will fulfill the

needs of the subordinate agency. In instances where the subordinate agency is unable to

receive adequate information regarding the measure, the opportunity to improve

performance is diminished. Sensitivity to agency discretion must therefore be applied

across numerous levels of governance which in turn dilutes the impact of the federal need

for information. Without appropriate incentives or resource assistance, the development of

reliable and valid data that meets the needs of all stakeholders represents a challenge.

Coffey et al. (2008) identify several challenges in the collection of mental health and

substance abuse data and identify practical policy solutions to improve data quality. The

most significant barrier identified was improving the interoperability of datasets

(Coffey,2008). One of the most significant challenges specific to mental health and

substance abuse treatment is the issue of confidentiality and the use of Health Information

14

and Portability and Accountability Act (HIPAA) compliant client identifiers (Coffey et al.,

2008). Another potential challenge to data is not having the technological ability to track

innovations. Changes at the professional or federal level are difficult to track in the legacy

systems used by state agencies.

Performance information regarding agency performance can be implemented with

positive results even if the information is not ideal (Coffey, 1999). However, the

information needs to be involved in a continuous improvement process with dual feedback

loops. One of the influencing contextual factors is the influence of goal ambiguity on

performance measurement. In Chun and Rainey’s (2005) descriptive study of goal

ambiguity at federal agencies, the author’s highlighted the need for efforts to increase goal

clarity to account for agency differences (p.24). Hall and Handley (2011) used a national

survey of city government officials and found mixed results regarding the implementation

of federal performance measures at the local level. Performance measures are likely to

provide incomplete but sufficient information regarding the specified domain of use. The

challenge resides in using the data for multiple purposes beyond the initially intended use.

Given resource limitations, flexibility in the use of performance measures provides

agencies with the ability to meet the needs of multiple stakeholders.

Impact on State

Challenge of Multiple Levels Performance Measures

15

The performance data collected for multiple agencies creates difficulty in making

management decisions at the state level. While there are tertiary benefits in stakeholder

accountability, it serves to provide a complex message to agencies trying to initiate change

especially when the agency has multiple stakeholders at the federal, state, and local levels.

In the case where there are related policies it becomes difficult for an agency to prioritize

and measurement decisions may be the result of responding to the agent that is most

actively engaging the agency. There are other reporting mechanics that may influence the

reporting of the agency such as the presence of an incentive, the frequency of reporting,

and intensity of reporting serve to impact the quality of the performance measure. Courty

and Marschke (2007) use the federal job training program to explain the necessary

feedback loop between performance measure development and agency reporting. The

process requires a continual process improvement for performance measures which

incorporates the responses to the performance measures. In addition to these

measurements of reporting intensity there are also factors related to the type of report.

One possible sustainable solution to the multiple levels of performance measures is the use

of performance information capture to meet the accountability requirements of one level of

the intergovernmental system at separate intergovernmental level.

Challenge of Missing Data in Performance Measures

The results of this study highlight missing data as one of the major barriers to the

states using federal performance measures to benchmark performance with other states.

Federal performance measures serve as a measure on the use of federal expenditures.

However, performance measures serve as an important resource for the development of

16

performance management systems and benchmarking at the state level. In order to be

useful to the states, performance measures need to provide accurate and reliable data from

which states can use the data for national comparisons. For most of the population served,

the state agency is removed from the process of service provision and while it has access to

the Medicaid billing data, information regarding key management decisions resides at the

level of program management. There is a delicate balance between the amounts of data and

the state authority requires meeting contract requirements and the amount of extra

information regarding performance. Performance measures can be integrated into the

contract but the inclusion of measures requires additional dedication of resources and

political negotiation.

Data

The Substance Abuse and Mental Health Services Administration developed the National

Outcome Measures and developed a national collection of outcome measures developed

along ten domains. The ten domains are associated with related outcomes and measures

that represent mental health, substance abuse. Substance abuse measures are separated

into treatment and prevention categories. The broad coverage of the domains across the

behavioral health spectrum creates a situation where the measures are not applicable

across all practice domains. Using information from this national database for FY 2007 and

FY 2009, the impact of national efforts on evidence-based practices will be analyzed. The

goal is to identify the interaction between Oregon’s efforts to monitor and increase the

amount of evidence-based practices at the state and county-level with the reporting on

national level outcome measures.

17

Reviewing the performance of 50 states using 2007 and 2009 national outcome

measures for adult and children’s evidence-based practice implementation provides

information on the reporting levels of various state mental health systems. The results of

this analysis identify areas that highlight incongruences in information uptake at the state

and federal level. There is a demonstrated lack of reporting of practices at the state-level.

There are ten evidence-based practice measures contained in the National Outcome

Measures (NOMs). Seven of these measures are for the adult population, and three are for

the children’s population. The evidence-based practices were identified through a

professional research consensus panel convened by Robert Woods Johnson Foundation in

1998 (SAMSHA, 2008). The practices identified by the panel included six practices: Illness

Management and Recovery, Supported Employment, Family Psycho-education, Assertive

Community Treatment, Integrated Treatment for Co-Occurring Disorders, and Medication

Management are represented in the NOM measures (SAMSHA, 2008). In addition to

tracking these measures, SAMSHA provides free implementation information for each of

the practices. These practices have been shown to be impacted by contextual factors that

limit their implementation such as the financing and regulation of individual practices as

well as leadership and training and quality issues (Isett et al., 2007). There are also

variations in the implementations of individual evidence-based practices. Given the level of

variability in the system, it is difficult to standardize implementation across state agencies.

This variation is reflected in the reported URS data. The results of the analysis is the level of

variability in the use of innovative practices and serves as a signal to states that results

based on the implementation of all tracked evidence-based practices might be muted at

best. The lack of reliable reporting across states provides states with general information

18

regarding the tracking of evidence-based practices for a state performance management

system. The information is based on the level of report and not the levels reported in the

data. Stated differently, the states gain information on the capacity of other states to report

and the level in which this innovation needs to be a priority. Lacking federal incentive, the

states learn the baseline for report and can make administrative decisions based on this

information. In the case of Oregon, the state mandate provided a mechanism in which to

focus on evidence-based practices from a more general approach and not focus on the six

evidence-based practices required for the Uniform Reporting System.

The amount of data missing in the Uniform Reporting System across states for the

ten identified evidence-based practices is significant and limits the analysis available to

state decision-making systems. In 2007 and 2009 the State of Oregon reported on half of

the ten reported evidence-based practices. While the challenges of missing data preclude

any complicated analysis they do provide information to state performance management

systems regarding the level of tracking and implementation of selected evidence-based

practices across state mental health agencies. The agency can make administrative

decisions regarding the amount of investment in the implementation of national evidence-

based practices and instead invest resources into the state mandate to track the amount of

funding on evidence-based practices. Given that state mental health agencies operate in a

resource limited environment and are subject to goals from different levels of the

intergovernmental system, this information may be important in making decisions on

resource allocation.

Frequency Distributions

19

The frequency distributions for the reported evidence-based practices for the years

2007 and 2009 were analyzed using STATA 11.0. Supported Employment and Assertive

Community Treatment had 27 instances of missing data; Supported Housing displayed 32

instances out of 102 observations of missing data for the years 2007 and 2009. Family

psycho-education and medication management had 75 instances of missing data; Illness

management had 69 instances of missing data. Dual Diagnosis treatment experienced 56

instances and Therapeutic Foster Care experienced 52 instances of missing data. Multi-

Systemic Therapy experienced 69 instances of missing data. Functional Family care

experienced 79 instances of missing data.

The goal of these measures was to increase the use of these evidence-based

practices so it is assumed that the practices would have low initial representation that rises

over time. The challenge encountered was the extent of missing data across all measures

and states for 2007 and 2009. Louisiana, Tennessee, and Washington were the only states

to report on all of the evidence-based practice measures which occurred in 2009. The state

of Wyoming did not report on any of the performance measures for 2007 and 2009. Alaska,

California, Louisiana, and Mississippi did not report on any of the evidence-based practice

measures in 2007. The level of missing data reflects a general lack of reporting, tracking,

and implementation of evidence-based practices across states over the 2007 to 2009 time

period.

SAMSHA has addressed issue in data quality experienced in the URS data. In 2007,

SAMHSA provided up to $142,200 per year in total costs to states for the purpose of

improving data quality (SAMSHA, 2007). One of the possible improvement opportunities

20

identified was the use of web based mechanisms to increase reporting (SAMSHA, 2007).

There are many challenges to the use of the Uniform Reporting System database as a

function of performance management. One of the biggest challenges is the issue of non-

report on measures. Given that non-response is more highly attributed to a lack of report

missing values can be categorized as Non-Ignorable (NI) error. Non-ignorable missing data

can be modeled but require significant information regarding the parameters of the

missing data (Luke, et al., 2002). Lacking information on the data parameters, the database

is of limited use. The importance of the National Outcome Measures is the ability to track

progress on particular program aspect with other states. The inability to track across states

limits the use of the national outcome measure system.

Oregon Specific Implementation

Descriptive data for the adoption of evidence-based practices for the years 2007 and

2009 for the state of Oregon reveal trends in reporting of evidence-based practices.

Medications Management is the only practice with a reported value over 1%. Medications

Management grew from 27.94% in 2007 to 34.3% in 2009. Several of the practices had no

report for any of the three years. There was a considerable amount of indication of a lack

of report of evidence-based practices. Of total 1,530 observations over a three year period,

825 (54%) of the reports had no reported data. The highest level of report was for

Functional Family Therapy with 118 out of 153 observations. The second most non-

reported evidence-based practice was Illness Self-Management with 111.

21

Discussion

While the National Outcome Measures provided information regarding state

progress in the tracking of the six national evidence-based practices, they do not capture

state efforts to increase the use of evidence-based practices. The six national evidence-

based practices were derived through an expert consensus practice in 1998, and they are

not sensitive to improvements in determining evidence-based practices overtime. There

are some inconsistencies at the federal agency level. SAMSHA maintains the National

Registry of Evidence-Based Programs and Practices (NREPP) which rates the effectiveness

of evidence-base practices along multiple domains. The NREPP effort is similar to the State

of Oregon process in which evidence-based practices are submitted by providers and

assessed based on the available research. Lacking a consensus on the specific practices

employed at the state levels there is the need to seek additional information. This process

allows the agency to gain information on the current practices and monitor the influx of

new practices.

As of July 27, 2011, the NREPP has tracked over 200 practices (SAMSHA, 2011).

Where the National Outcome Measures provide a specific focus on particular practices,

they are not sensitive to the overall proliferation of evidence-based practices. A state may

increase performance on the narrowly defined evidence-based practice measures but have

limited impact overall on the mental health service delivery process. The greatest challenge

to the tracking of evidence-based practices at the national level is the lack of data capacity

related to tracking the use of the particular evidence-based practice.

22

Variations in Implementation

The differences in the reporting at the federal and state level provide different

pictures of the same policy implementation. The federal performance monitoring focuses

on the amount of the population receiving a particular service, the Oregon evidence-based

practice implementation tracked the agency level of adoption. There are advantages and

disadvantages to both approaches. The Oregon implementation model highlights local

contextual factors that impact the interpretation of criteria in performance monitoring. At

the local level, evidence-based practices were interpreted widely and developed through a

consensus process. At the national level, a narrowly defined standardized criterion was

adopted. The goal of the SAMHSA URS is to provide accountability to congress on mental

health spending. The impetus for the Oregon implementation was the state-legislature

which focused its concern on the amount of state funds purchasing evidence-based

practices. In both cases, the respective legislature guided the performance measurement

system. One of the possible advantages available to the state agency is to use both sets of

information to develop a performance management system that monitors both state and

federal requirements.

Potential Improvements

One promising development is the use of electronic health records which can

increase the transmission of data from the provider to the monitoring agency

automatically. Additional protections such as incentives for performance on measures are

an alternative to increase data reliability and report. While at the federal level it is

23

advantageous to have information on performance on a wide variety of measures, at the

agency level there is limited incentive to provide reliable data when there are competing

demands for data report.

While the state of Oregon did not track all of the evidence-based practices for the

URS, the state developed a separate process for tracking the implementation of all

evidence-based practices. In 2005, Oregon had identified 80 evidence-based practices. In

2008, Oregon tracked 110 evidence-based practices. In 2005, the most reported evidence-

based practice was Motivational Interviewing with a 75 counties identifying that the

practice was purchased. The Oregon survey used a different unit of analysis compared to

the SAMSHA URS report. The Oregon survey focused on county purchasing of an evidence-

based practice where the SAMSHA URS database tracks the percent of individuals that

received a particular service.

National Performance Management Guidelines

The National Performance Management Advisory Commission provides a

performance management framework for state and local government. The focus of this

framework is to provide general guidelines for the development of public performance

management systems. The framework focuses on a continual cycle of development which

involves stakeholders and standard principles in the development of performance

management systems that address accountability and the capacity for learning and

improvement (National Performance Management Advisory Commission, p.1). This

framework involves the normative performance management theories. These management

theories focus on the use of information in decision-making but remain unclear on the

24

practicality of monitoring and tracking data with reduced resources and third-party

governance structures. In instances where federal performance measures do not overlap

with state measures, states may make the rational decision and invest resources with the

stakeholder of greatest influence. Performance measures that are not a core element of the

mission have less chance of completion. Accountability professionals play an important

role in assisting agencies in developing performance measure systems that interact with

federal systems and provide useful baseline information that improves the functioning of

the agency. The decision to delegate resources to collection of performance measures is a

management decision which accounts for the transaction cost of the monitoring effort.

Accountability professionals can provide guidance in developing performance management

systems at the state agency level that allow state agencies to utilize existing data collection

systems and integrate these mechanisms to increase the information available for

managerial decisions. Performance data that serves the function of reporting to the

congress is not necessarily effective to the state agency. The most important level of

information in which program decisions can be made rests at the interface between the

provider and the state agency. State comparisons are effective for the state agency to make

decisions about provider practices but require sufficiently valid and reliable data.

25

References

Behn, R. D. (2003). Why Measure Performance? Different Purposes Require

Different Measures. Public Administration Review, 63(5), 586-606.

Bird, S. M., Sir David, C., Farewell, V. T., Harvey, G., Tim, H., & Peter C., S. (2005).

Performance indicators: good, bad, and ugly. Journal of the Royal Statistical Society: Series A

(Statistics in Society), 168(1), 1-27.

Brolin, M., Seaver, C., & Nalty, D. Performance Management: Improving State Systems

through Information-based Decision making. DHHS Publication No. 05-3983. Rockville, MD:

Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services

Administration, 2004.

Chun, Y. H., & Rainey, H. G. (2005). Goal Ambiguity in U.S. Federal Agencies. Journal

of Public Administration Research and Theory, 15(1), 1 -30.

Coffey, R. M. (1999). Case Mix Information in the United States: Fifteen Years of

Management and Clinical Experience. CaseMix Quarterly, 2(3), 7-16.

Coffey, R. M., Buck, J. A., Kassed, C. A., Dilonardo, J., Forhan, C., Marder, W. D., &

Vandivort-Warren, R. (2008). Transforming Mental Health and Substance Abuse Data

Systems in the United States. Psychiatric Services, 59(11), 1257-1263.

Courty, P., & Marschke, G. (2007). Making Government Accountable: Lessons from a

Federal Job Training Program. Public Administration Review, 67(5),

26

Derthick, M. (1970). The influence of federal grants: public assistance in

Massachusetts. Harvard University Press.

Dubnick, M. J., & Frederickson, H. G. (2009). Accountable Agents: Federal

Performance Measurement and Third-Party Government. Journal of Public Administration

Research and Theory, 20(Supplement 1),

Edmunds, M., Frank, R., Hogan, M., McCarty, D., Robinson-Beale, R., and Weisner, C.

(Eds); Committee on Quality Assurance and Accreditation Guidelines for Managed

Behavioral Health Care, Institute of Medicine. (1997). Managing Managed Care: Quality

Improvement in Behavioral Health. Washington, D.C.: The National Academies Press.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of Innovations

in Service Organizations: Systematic Review and Recommendations. The Milbank Quarterly

2004, 82:581-629.

Ganju V: Mental Health Quality and Accountability: The Role of Evidence-Based

Practices and Performance Measurement. Administration and Policy in Mental Health and

Mental Health Services Research 2006, 33:659-665.

Gunsteren, H. R. van. (1976). The quest for control: a critique of the rational-central-

rule approach in public affairs. London: Wiley.

Halachmi, A. (2002). Performance measurement: a look at some possible

dysfunctions. Work Study, 51(5), 230-239.

Halachmi, A. (2005). Performance measurement: test the water before you dive in.

International Review of Administrative Sciences, 71(2), 255 -266.

27

Hall, J., & Handley, D. (2011). City Adoption of Federal Performance Measurement

Requirements. Public Performance & Management Review, 34(4), 443-466.

Ho, A. (2007). The Governance Challenges of the Government Performance and

Results Act A Case Study of the Substance Abuse and Mental Health Administration. Public

Performance & Management Review, 30(3), 369-397.

Ingraham, P. W., Joyce, P. G., & Donahue, A. K. (2003). Government performance: why

management matters. Baltimore: The Johns Hopkins University Press.

Isett, K. R., Burnam, M. A., Coleman-Beattie, B., Hyde, P. S., Morrissey, J. P.,

Magnabosco, J., Rapp, C. A., et al. (2007). The state policy context of implementation issues

for evidence-based practices in mental health. Psychiatric Services (Washington, D.C.), 58(7),

914-921.

Jacobsen, K., & McGuire, T. G. (1996). Federal Block Grants and State Spending: The

Alcohol, Drug Abuse, and Mental Health Block Grant and State Agency Behavior. Journal of

Health Politics, Policy and Law, 21(4), 751-768.

Julnes, P. D. L., & Holzer, M. (2001). Promoting the utilization of performance

measures in public organizations: An empirical study of factors affecting adoption and

implementation. Public Administration Review, 61(6), 693-708.

Light, P. C. (1997). The tides of reform: making government work, 1945-1995. Yale

University Press.

28

Lilford, R., Mohammed, M. A., Spiegelhalter, D., & Thomson, R. (2004). Use and

misuse of process and outcome data in managing performance of acute medical care:

avoiding institutional stigma. The Lancet, 363(9415), 1147-1154.

Luke, D. A., Marsh, L., Smithson, M., Allison, P. D., Chen, P. Y., Best, S. J., Rudas, T., et al.

(2001). Missing data. SAGE.

Mitchell, Sandra and Brown, Ann. Performance Measurement Data: Is It Fit For Use?

Data Quality Initiatives At CIHI. Proceedings of Performance Measurement in Healthcare:

Responding to the Need for Greater Accountability. Toronto, Ontario, Canada. May 15-16,

2002.

Manderscheid, R. (2006). Promoting Self-Assessment and Accountability:

Introduction to Quality Improvement Measures In Center for Mental Health Services. Mental

Health, United States, 2004. Manderscheid, R.W., and Berry, J.T., eds. DHHS Pub no. (SMA)-

06-4195. Rockville, MD: Substance Abuse and Mental Health Services Administration.

Mays GP, McHugh MC, Shim K, Perry N, Lenaway D, Halverson PK, Moonesinghe

R.(2006). Institutional and economic determinants of public health system performance.

American Journal of Public Health. Mar;96(3):523-31.

Mays GP, Smith SA, Ingram RC, Racster LJ, Lamberth CD, Lovely ES (2009). Public

health delivery systems: evidence, uncertainty, and emerging research needs. American

Journal of Preventative Medicine. Mar;36(3):256-65.

29

National Association of State Mental Health Program Directors Research Institute,

Inc. (2005). Results of a survey of state directors of adult and child mental health services on

implementation of evidence-based practices. Alexandria, VA.

National Association of State Mental Health Program Directors Research Institute,

Inc. (2009). User Guide: Compiling and Reporting Client Level Data For The Mental Health

NOMS (For use by pilot States),Version 4.0, July 2009. Last Accessed on 8/09/2011 at

http://www.nri-inc.org/projects/SDICC/forms/Client_Level_Data_Users_Guide_1-14-

10.pdf .

National Association of State Mental Health Program Directors Research Institute,

Inc. (2011). State Data Infrastructure Coordinating Center (SDICC). Retrieved August 5,

2011, from http://www.nri-inc.org/projects/SDICC/

National Performance Management Advisory Commission (2010). A Performance

Management Framework for State and Local Government: From Measurement and Reporting

to Management and Improving. Last Accessed on 7/27/2011 at

http://www.pmcommission.org/

New Freedom Commission on Mental Health. (2003). Achieving the promise:

transforming mental health care in America. Final report. (DHHS Pub. No. SMA-03-3832).

Rockville, MD.

Oregon Revised Statute. 182.525 Available at:

http://www.leg.state.or.us/ors/182.html

30

Perrin, E.B.,Koshel, J.J. Editors; Panel on Performance Measures and Data for Public

Health Performance Partnership Grants, National Research Council. (1997). Assessment of

Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington,

D.C.: The National Academies Press.

Pidd, M. (2007). Perversity in public service performance measurement. Business

Performance Measurement (Second edition). Cambridge University Press.

Pipino,L. L., Lee,Y., W., and Wang, R. Y. 2002. Data quality assessment. Commun. ACM

45, 4 (April 2002), 211-218.

Posner, P. L. (2002). Accountability challenges of third-party government. In The

tools of government: A guide to the new governance, ed. L. M. Salamon, 523–51. New York,

NY: Oxford Univ. Press.

Power, M. (1997). The Audit Society: Rituals of Verification. Oxford: Oxford

University Press.

Radin, B. A. (2000a). Intergovernmental Relationships and the Federal Performance

Movement. Publius, 30(1), 143-158.

Radin, B.A. (2000b). The Government Performance and Results Act and the tradition

of federal management reform: Square pegs in round holes? Journal of Public

Administration Research and Theory, 10(1), 111–135.

Rieckmann, T. R., Kovas, A. E., Fussell, H. E., & Stettler, N. M. (2008). Implementation

of Evidence-Based Practices for Treatment of Alcohol and Drug Disorders: The Role of the

State Authority. The Journal of Behavioral Health Services & Research, 36(4), 407-419

31

Rubenstein, R., Schwarfz, A. E., & Stiefel, L. (2003). Better than Raw: A Guide to

Measuring Organizational Performance with Adjusted Performance Measures. Public

Administration Review, 63(5), 607-615.

Substance Abuse and Mental Health Services Administration (2008). Assertive

Community Treatment: How to Use the Evidence-Based Practice KITs. DHHS Pub. No. SMA-

08-4344, Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental

Health Services Administration, U.S. Department of Health and Human Services

Substance Abuse and Mental Health Services Administration. (2011a). SAMHSA

Block Grants. Last Accessed on August 5, 2011, from

http://www.samhsa.gov/grants/blockgrant/

Substance Abuse and Mental Health Services Administration. (2011). NREPP. Last

Accessed on August 5, 2011, from http://nrepp.samhsa.gov/

Substance Abuse and Mental Health Services Administration. (2011b). Leading

Change: A Plan for SAMHSA’s Roles and Actions 2011-2014. HHS Publication No. (SMA) 11-

4629. Rockville, MD: Substance Abuse and Mental Health Services Administration.

SAMHSA 2009 Uniform Reporting System (URS) Output Tables. (n.d.). . Retrieved

August 5, 2011, from http://www.samhsa.gov/dataoutcomes/urs/urs2009.aspx.

Substance Abuse and Mental Health Services Administration. (2007). $21 Million

Awarded to Help States Enhance Mental Health Data Recording and Reporting Systems. Press

Release 9/26/2007. Last accessed on 8/6/2011.

http://www.samhsa.gov/newsroom/advisories/0709265213.aspx

32

U.S. Department of Health and Human Services. (1999). Mental Health: A Report of

the General—Executive Summary. Rockville, MD: U.S. Department of Health and Human

Services, Substance Abuse and Mental Health Services Administration, Center for Mental

Health Services, National Institutes of Health, National Institute of Mental Health.

United States. General Accounting Office. GAO/HEHS-95-74 Block Grants Health,

Education, and. Human Services Division. B-260302. February 9, 1995

U.S. General Accounting Office, Managing for Results: Agency Progress in Linking

Performance Plans with Budgets and Financial Statements (GAO-02–236) (Washington, DC:

U.S. Government Printing Office, 2002 (ADD)

Westphal, J.D., R. Gulati, and S.M. Shortell. 1997. Customization or Conformity? An

Institutional and Network Perspective on the Content and Consequences of Total Quality

Management Adoption. Administrative Sciences Quarterly 42(366):394.

Wildavsky, A. (1974). The Politics of the Budgetary Process. Boston; Little Brown

and Company. Behn, R. D. (2003). Why Measure Performance? Different Purposes Require

Different Measures. Public Administration Review, 63(5), 586-606.

Wright, D. S. (1988). Understanding Intergovernmental Relations (3rd ed.). Pacific

Grove,CA: Brooks Cole.