Roberts/Rosenberg/Dever Presentation

1

Performance Management for Justice Information Sharing

David J. RobertsGlobal Justice Consulting

Linda RosenbergOffice of Criminal Justice System Improvements, PCCD

Michael DeverState Policy Advisory, BJA

2006 BJA/SEARCH Regional Information Sharing Conference

February 6, 2007 Houston, Texas

2

“Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”

—H. James Harrington

3

WHY evaluate performance?

• Information is control• Provides feedback to improve program

performance• Provides information for resource

allocation• Enables effective planning• Tests generalizations based on

experiences and assumptions• Market and develop support among

funding bodies, constituents, and staff.

4

Process vs. Impact Evaluations

• Process evaluations focus on how the initiative was executed; the activities, efforts, and workflow associated with the response. Process evaluations ask whether the response occurred as planned, and whether all components worked as intended. Fundamentally, a process evaluation posits the question, “Are we doing the thing right?”

• Impact evaluations focus on the outcome (the what) of the initiative; the output (products and services) and outcome (results, accomplishment, impact). Did the problem decline or cease? And if so, was the response the proximate cause of the decline? Fundamentally, the impact evaluation posits the question, “Are we doing the right thing(s)?”

5

Balanced Scorecard

Originally developed in business by Kaplan & Norton

1. Financial – How do we look to stakeholders?

2. Customer – How well do we satisfy our internal and external customers’ needs?

3. Internal Business Process – How well do we perform at key internal business processes?

4. Learning and Growth – Are we able to sustain innovation, change, and continuous improvement?

6

Balanced Scorecard for Law Enforcement (Mark Moore, et al)

1. Reduce criminal victimization2. Call offenders to account3. Reduce fear and enhance personal

security4. Guarantee safety in public spaces5. Use financial resources fairly, efficiently,

and effectively6. Use force and authority fairly, efficiently,

and effectively7. Satisfy customer demands/achieve

legitimacy with those policed

7

Trial Court Performance Standards

1. Access to Justice;

2. Expedition and Timeliness;

3. Equality, Fairness and Integrity;

4. Independence and Accountability; and

5. Public Trust and Confidence.

8

Universal IJIS Elements

• Definition: The ability to access and share critical information at key decision points throughout the whole of the justice enterprise.

• Scope: Recognition that the boundaries of the enterprise are increasingly elastic—engaging not only justice, but also emergency & disaster management, intelligence, homeland security, first responders, health & social services, private industry, the public, etc.

• Goal: Get the right information, to the right people, all of the time—underscores the need for dynamic information exchange.

9

Information Sharing Objectives

• What is the problem we’re addressing?• What information do we have regarding

current levels of performance?• What is it that we’re trying to do?• 3 Universal Objectives:

– Improve Public Safety and Homeland Security;– Enhance the Quality and Equality of Justice;– Gain Operational Efficiencies, Effectiveness, and

demonstrate Return on Investment (ROI).

10

Public Safety Measures

• Increase the percentage of court dispositions that can be matched to an arrest—this will improve the quality of the computerized criminal history records

• Decrease the average response time to establish a positive identification following an arrest

• Reduce the number of incidents of criminal records being associated with the wrong person

• Reduce recidivism• Decrease the amount of time it takes to serve a warrant• Decrease the amount of time for law enforcement to have

details on protection orders.• Reduce the fear of crime in target neighborhoods• Reduce the amount of time it takes users of the integrated

justice system to respond to a request from the public• Reduce the time it takes to complete a criminal history

background check• Reduce the number of agencies that can’t communicate

with each other.

11

Quality of Justice Measures

• Reduce the number of civilian complaints against local law enforcement

• Reduce the number of continuances per case that result from scheduling conflicts between the courts, law enforcement, and prosecution

• Reduce the number of cases without a next scheduled event

• Reduce the average number of days or hours from arrest to arraignment

• Reduce the average time a defendant is held while waiting for a bond decision

• Reduce the time it takes for correctional facility intake• Reduce the number of days it takes to process cases from

arrest to disposition• Reduce the number of false arrests.• Reduce the amount of missing data.

12

Efficiency/Effectiveness Measures

• Reduce the number of hours that staff spend entering data manually or electronically

• Reduce the costs of copying documents for justice organizations

• Reduce the number of hours spent filing documents manually

• Reduce the number of hours spent searching other governmental databases

• Increase the number of law enforcement personnel performing community policing tasks, instead of administrative tasks

• Increase the number of electronic data transfers between justice agencies

• Reduce the amount of missing information in criminal justice databases

• Reduce the number of corrections needed in databases maintained by CJIS agencies

• Decrease the number of warrants that never get entered into the state registry

• Increase the number of query hits on each agency database

• Reduce the number of hours it takes to enter a court disposition into the state criminal history repository

13

Problem Identification & Objective

• Problem—Dispositions in many felony cases are taking a significant amount of time.– So what? Baseline metrics assessing the operation of the

justice system demonstrate the following effects:• Increased costs—more people held in confinement

for longer periods of time, more public support for families, etc.

• Fewer convictions—witnesses disappear, forget, etc.• Decrease in confidence in the justice system—

victims and the general public wonder what’s going on…why can’t the system operate more efficiently

• Objective:– Reduce the time to disposition of felony cases to an

average of 6 months statewide within 12 months after implementation.

14

Walking Thru an Example

15

Specific Example

16

Critical Assumptions

• Baseline data regarding current or historical performance of the system

• Access, ability and willingness to capture data regarding on-going performance

• Timely, accurate and complete data collection

• Appropriate and sufficiently detailed analysis techniques

• Staff to conduct the analysis and reports.

17

Case Study – PA County Integration

• One of the primary goals of the PCCD is to enhance public safety through collaboration and information sharing.

• To accomplish this goal, PCCD designed a public safety strategy that combines county planning with technology solutions.

• PCCD supports county-planning activities which encourage decision-makers to view the justice system from a holistic perspective rather than from their individual fiefdoms. These efforts are supported with financial resources and technical assistance.

• This holistic perspective and leadership from county executives

is critical for improved county decision-making as well as for the implementation of integrated justice systems.

18


• PCCD educated participating county leadership teams on the value of information sharing, from a policy perspective, public safety perspective, and operational efficiency perspective. This helped ensure support and buy-in from leadership.

• PCCD educated them on a technical strategy that would allow them to maintain their autonomy as well as provide them with an executive dashboard of timely and accurate information critical for improved decision-making.

• To prove our technical strategy, PCCD designed a pilot effort

that tests and documents the effectiveness of a distributed approach to information sharing which utilizes SOA, web services, and GJXDM.

19


The strategy defined the following goals: • Implementation of a technical solution in small, medium, and

large counties that respects the autonomy of participating agencies

• Create efficiencies through reduced data entry and increased timeliness of information

• Enhance public safety through more accurate and timely information

• Leverage existing systems including JNET, county records management systems and other state systems

• Exchange data horizontally and vertically between state and county systems

20


The following tangible results were achieved as a result of our goals:

• Reduced data entry, and clerical and record-keeping activity, as a result of information being automatically transferred from the sender’s files to the receiver’s files.

• Less time lost in actions delayed while waiting for information to be transferred and records updated through document transfers, or through batch file transfers.

• Reduced filing costs due to paperless, media-less information exchanges

21


The following tangible results could be achieved as a result of one message being implemented in the counties:

Potential statewide annual cost avoidance to probation departments for data entry = (15,055 hours at an estimated labor cost of $9 per hour) = $135,495.00

Potential statewide annual cost avoidance to jails for data entry (8,008 hrs x $9 per hr) = $72,072

22


The following intangible results were achieved:• Increased accuracy of data due to fewer manual records

transcription and key data entry steps, fewer misplaced information transfer forms.

• More complete information can be maintained, since additional data can be transferred without additional costs for transcription or key data re-entry.

• Information can be extracted from operational information exchanges, summarized and stored for business measurements and trend analysis, avoiding costly, time consuming collection and analysis of often-incomplete historic data.

• On-going support of information sharing activities from county state executives.

23

Case Study – Victim Compensation

One of the primary goals of the PCCD, is to serve individuals who experience criminal victimization in Pennsylvania. Often, victims must deal with financial hardships they face as a result of their victimization. Thus, the PCCD Victims Compensation Assistance Program was established to ease the financial burden victims may face as a result of a crime by providing them with critical financial support in their time of need. This financial support is provided to victims after the submission, processing, and approval of a complicated claim paperwork process.

24


In 2002, the processing time for a claim, or the time it took to receive financial assistance from PCCD, was over 26 weeks! Today, it's less than nine weeks.

In 2006, 7000 claims were processed resulting in $12 million being distributed to crime victims.

These improvements were achieved through five years of incremental automation enhancements that were planned, implemented, quantified, and justified to state and federal funding sources.

25


Some of the performance measures,which justified the on-going expansion of this system, entitled DAVE, include:

1) The processing time for claims continued to drop (26 weeks to 12 weeks to 9 weeks and dropping).

2) The DAVE system has enabled PCCD to increase the number of claims in which it processes by 200% with out having to increase staff resources.

3) Over 250 advocates were trained on the use of the DAVE system. This has resulted in 30% of the claims coming in electronically from individuals in the field working directly with crime victims. This saves PCCD staff time, ensures that victims has access to critical services, and ultimately ensures that they receive financial support in a more timely fashion.

26


4) In 2005, an automated electronic interface between PCCD, the comptroller and the treasurer was developed. This interface resulted in the elimination of over 100,000 documents that were data entered and processed by three different agencies. This has lead to huge cost savings for all three agencies and reduced the likelihood of error by 100%. In addition, the experiences learned in developing this interface, will be applied to other PCCD systems including our E-grants system.

27


5) In 2006, a web-based interface to DAVE was developed. This interface allows claimants and service providers to check the status of their claims on-line and with out human intervention. This interface has reduced the number of phone calls to staff by 20% while the number of claims actually increased by 26% over the same period.

6) PCCD will be the first victim compensation program in the nation to allow claimants to file on-line via the web. It anticipated that this enhancement will result in additional cost savings, efficiencies, and improved services to victims of crime in PA.

28

Managing Performance

“Performance measurement, in simplest terms, is the comparison of actual levels of performance to pre-established target levels of performance. To be effective, performance must be linked to the organizational strategic plan.

Performance-based management essentially uses performance measurement information to manage and improve performance and to demonstrate what has been accomplished.

In other words, performance measurement is a critical component of performance-based management.”

Source: Will Artley, D.J. Ellison and Bill Kennedy, The Performance-Based Management Handbook, Volume 1:

Establishing and Maintaining a Performance-Based Management Program (Washington, DC: U.S. Department of Energy,

2001), p. 4.

29

Performance Management Paradigm

The Six Steps to Establishing a Performance-Based Management Program

Source: Will Artley, DJ Ellison and Bill Kennedy, The Performance-Based Management

Handbook, Volume 1: Establishing and Maintaining a Performance-Based Management

Program (Washington, DC: U.S. Department of Energy, 2001)

30

Step 1: Define Mission and Strategic Performance Objectives1. Mission statements identify the overall purpose for which

the organization is organized. 2. Vision statements describe the future business

environment and the role of the organization within it. 3. Value statements reflect fundamental beliefs and values

guiding the agency, the nature of their responsibilities, and the philosophy underlying their approach.

4. Assumptions are also frequently discussed in strategic planning efforts, describing business environmental conditions that are expected in the future.

5. Business strategies identify how objectives are to be accomplished, e.g., community-oriented policing, integration of justice information systems, subscription/notification capabilities, etc.

31

Step 2: Establishing an Integrated Performance Management FrameworkMajor Elements in Creating a Performance Management

Framework

1. Define the Relationship of Performance Measurement to the Strategic Planning Process

2. Build the Performance Management Team3. Address Stakeholder/Customer Needs4. Understand Performance Measurement Terminology5. Manage Performance Measurement6. Accept Accountability for Measures 7. Communicate8. Know How to Check/Test Your Measures 9. Learn From Others 10. What Do You Measure Yourself Against

32

Performance Planning Template• Performance Plan Template Defined/Action taken • Mission/Vision/Values/Assumptions Relate project to Agency Mission/Vision, etc.• Strategic Performance Objective 1 Defined• Operation/Activity Title• Purpose Describe purpose(s) of initiative• Execution Define how you’re going to do it• Performance Target(s) Identify target(s) (e.g., reduce violent crime by 10%)• Performance Measures How you’re going to measure it• Owner Person responsible/accountable• Resources Resources needed for this initiative• Budget Funding dedicated to this initiative• FTE Staffing dedicated to this initiative

• Strategic Performance Objective 2 Defined• Operation/Activity Title• Purpose Describe purpose(s) of initiative• Execution Define how you’re going to do it• Performance Target(s) Target(s) (e.g., reduce violent crime by 10%)• Performance Measures How you’re going to measure it• Owner Person responsible/accountable• Resources Resources needed for this initiative• Budget Funding dedicated to this initiative• FTE Staffing dedicated to this initiative

33

Step 3: Establish Accountability for Performance

• Building Accountability:– Authority refers to the power to influence or

command thought, opinion, or behavior. – Responsibility means that one is liable to be called to

account as the primary cause, motive, or agent.– Accountability, on the other hand, is an obligation or

willingness to accept responsibility and to account for one's actions.

• Internal vs. External Accountability– Internal Organizational Accountability – Internal

organizational accountability refers to the establishment of the upward and downward flow of accountabilities between management and individuals and teams within the organization.

– External Organizational Accountability – In external organizational accountability, the organization answers to/reports to its stakeholders on both its organizational performance and organizational behavior.

34

Step 4: Establish a Process/System for Collecting Data to Assess Performance Develop a Plan:1. Information Requirements2. Information Sources3. Data Collection Processes4. Data Collection and Reporting Frequencies5. Data Collection Costs 6. Data Protection7. Data Quality8. Trial Run

35

Step 5: Establish a Process/System to Analyze, Review, and Report Data Data Analysis Strategies1. Assess the Quality of Data2. Employ Analytic Methods3. Calibrate Baseline Measures4. Test Hypotheses 5. Know What to Measure6. Data Presentation—How Will the Data be Used and

Reported?7. Create an Executive Dashboard

36

Sample Executive Dashboard

37

Step 6: Establish a Process/System to Use Information to Drive Improvement 1. Shape Organizational Culture2. Make Information Broadly Available3. Reengineer Business Processes4. Build Performance Management into Everyday

Operations a. Executive Support and Organizational Commitment

is Requiredb. Comprehensive System of Accountability and

Responsibility is Requiredc. Flattens the Organizationd. Identify Problems Earlye. Ensure Progress and Keep Projects on Target f. Demonstrate Value

38

Resources• Will Artley, DJ Ellison and Bill Kennedy, The Performance-Based

Management Handbook, Volume 1: Establishing and Maintaining a Performance-Based Management Program (Washington, DC: U.S. Department of Energy, 2001) at http://www.orau.gov/pbm/pbmhandbook/pbmhandbook.html

• John E. Eck, Assessing Responses to Problems: An Introductory Guide for Police Problem-Solvers (Washington, DC: Center for Problem-Oriented Policing, no date), at http://www.popcenter.org/Tools/tool-assessing.htm

• Michael Geerken, The Art of Performance Measurement for Criminal Justice Information System Projects, (Washington, DC: U.S. Department of Justice, Bureau of Justice Assistance, 2006 [forthcoming])

• Robert H. Langworthy (ed.), Measuring What Matters: Proceedings from the Policing Research Institute Meetings, (Washington, DC: NIJ/COPS, July 1999, NCJ 170610), pp. 37-53.

• David J. Roberts, Law Enforcement Tech Guide: Creating Performance Measures that Work! A Guide for Law Enforcement Executives and Managers, (Washington, DC: U.S. Department of Justice, Office of Community Oriented Policing Services, 2007 [forthcoming from SEARCH and COPS])

http://www.orau.gov/pbm/pbmhandbook/pbmhandbook.html

http://www.popcenter.org/Tools/tool-assessing.htm

39

Performance Measurement Resources

• More information on GPRA and PART can be found at:–www.omb.gov/part

–www.results.gov

–www.expectmore.gov

http://www.omb.gov/part

http://www.omb.gov/part

http://www.results.gov/

http://www.expectmore.gov/