Measuring Regulatory Excellence: Key Performance Indicators

2007 Annual Conference

Measuring Regulatory Excellence: Key

Performance IndicatorsBruce G. Matthews, P.Eng.

Professional Engineers Ontario

Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference

Overview• Project Definition and Scope

• What is a Key Performance Indicator?

• Project Process

• Challenges

• List of Potential KPIs

• Survey re: Relevance and Importance

• Recommended KPIs

• Aftermath


Project Definition and Scope

• Project arose from a Strategic Plan Objective: “Increase the public’s confidence in PEO as a regulator”

• Associated Strategic Initiative: “Develop and implement key performance indicators and targets of regulatory effectiveness, benchmarked against other comparable regulatory bodies.”


Project Definition and Scope con’t

• Literature review re: best practices

• Review of other regulators’ performance measurement practices

• Survey of stakeholders re: perceptions and expectations

• Assess availability of data for ongoing tracking / reporting of KPIs


What is a Key Performance Indicator?• A metric used to reflect the critical success

factors of an organization

• Can be financial or non-financial

• Nature of KPIs will differ depending upon the nature of an organization

• A KPI is a key part of a measurable objectives (which includes a direction, KPI, benchmark, target, timeframe)


Effective KPIs…• Should help define:

– The What – Are we doing the right things? What is our outcome in terms of impact and accomplishments?

– The How – Are we doing things right? Do we have the right inputs in terms of resources and processes?


Project Process• Literature Review

• Environmental Scan - Review performance measurement practices of other professional regulators

• Identify potential KPIs

• Survey stakeholders re: relevance and importance of potential KPIs

• Review pros and cons of potential KPIs


Literature Review re: Best Practices

• Performance measurement best practices– Balanced scorecard– UK National Consumer Council– “The Regulatory Craft”– “Red Tape to Smart Tape” International

Conference (2002)


Balanced Scorecard• Developed by Harvard professors Kaplan

and Norton

• Four categories– Customers– Internal Business Processes– Financial– Organizational Capabilities

• Weakness re: definition of “customer”


UK National Consumer Council

• Three steps to credible regulation– Objectives– Content– Structure and Governance

• www.ncc.org.uk


“The Regulatory Craft”• Book by Malcolm K. Sparrow (Brookings

Institution Press, 2000)

• Controlling risks, solving problems and managing compliance

• Matrix of “Regulatory Competencies” and “Business Results”

• Acknowledges “internal” versus “external” measures


Red Tape to Smart Tape Conference

• Sponsored by the Ontario Government in association with the Institute of Public Administration of Canada

• Core concept – Traditional business results measurements do not address public welfare issues and are not suitable for regulatory purposes

• www.smarttape.ca


Environmental Scan• Other engineering regulators in Canada

• Other professional regulatory bodies in Ontario and across Canada

• Most report key transactional statistics (e.g., number of licence applications)

• Little evidence of outcome measurement


Challenges• What does it mean to be an effective

regulator?

• Inability to benchmark due to lack of reporting by other “comparable regulatory bodies”

• Can’t use measures typical for professional associations

• KPIs must be evaluated in context


List of Potential KPIs• Competence programs

• Intrusions on regulatory mandate

• Complaints about licence holders or unlicensed individuals

• Licensing turnaround times

• Compliance action statistics

• Council evaluation


List of Potential KPIs con’t

• Fairness & dignity survey

• Complaints about PEO

• Employee engagement

• Percentage of licence holders with no reported violations

• Public confidence survey score

• Ratio of administrative overhead to program delivery cost


Survey re: Relevance and Importance

• PEO Council, Senior Management and selected external stakeholders

• Relevance – the applicability of a KPI as a measure of PEO’s effectiveness as a regulator

• Importance – the priority that should be given to a KPI as a measure of PEO’s effectiveness as a regulator


Survey re: Relevance and Importance

• Survey also included proposed short-term measures for each KPI, intended to cover the introductory transitional period

• Survey results were assessed in view of the best practices review– Need balance of internal and external

measures)– Need to ensure key regulatory competencies

are represented


Recommended KPIs• Competence programs

• Complaints about licence holders or unlicensed individuals

• Compliance action statistics

• Employee engagement

• Public confidence survey score


Competence Programs• Pros

– Competent members reflect a well regulated profession

– Competency is fundamental to professional regulation and easily understood by the public

• Cons– PEO currently has no competence programs

in place and none planned in the near term


Complaints about Licence Holders etc.

• Pros– Easily measured (transaction statistics

already reported)

• Cons– Frivolous or vexatious complaints could skew

the data– Does complaint volume really speak to

regulatory effectiveness?


Compliance Action Statistics

• Pros– Clear statement of enforcement activity– Some statistics already reported

• Cons– Need to define what qualifies as a

“compliance action”– Of greater interest to licence holders than to

the public


Employee Engagement• Pros

– Regular survey of employees already in place– Survey is widely recognized and used

• Cons– Is PEO willing to take action in deficient

areas?– Are the survey areas relevant to a

professional regulatory organization?


Public Confidence Survey Score

• Pros– Solid external measure of how PEO is

perceived– Survey would be conducted by an

independent research firm

• Cons– Frequency of measurement– Who is “the public”?


Implementation Issues• Currently no competence programs in

place at PEO

• Measurement data exists for only three of the recommended KPIs

• Public Confidence Survey still in development

• Issues of KPI reporting (timing, frequency, format) still to be determined


Aftermath• Project report received and considered by

PEO Council in November 2006

• Concern that recommended KPIs were all “lagging indicators” – there were no “leading indicators”

• Concern that recommended KPIs would drive policy and program development

• Concern about lack of competency programs


Aftermath con’t• PEO Council struck a task force to further

examine the question of KPIs for PEO

• Task force report (June 2007) recommends a new project with more research work re: critical success factors before defining KPIs

• Status as of September 2007: awaiting new project charter / terms of reference


Speaker Contact Information

Bruce G. Matthews, P.Eng.Manager, Complaints & DisciplineProfessional Engineers Ontario25 Sheppard Avenue West, Suite 1000Toronto, Ontario M2N 6S9T: 416-840-1076F: 416-224-9974E: [email protected]

2007 Annual Conference

Measuring Regulatory Excellence – A Self Audit Tool Focused on Entry to Practice

Jan RobinsonRegistrar & CEO

College of Physiotherapists of OntarioSeptember 8, 2007


Interesting times

• accountability

• transparency

• movement of internationally educated professionals

• public perspective


Politics in Ontario


Ontario Regulators for Access (ORA)

• Info Sharing

• Best practices

• Collaboration


“measures” ➪ government driven


report card to progress report


Key Factors Included• # of internationally trained applicants

(annual comparisons)• top five source countries• types of license granted• project initiatives

Focus on “10”


Thomson review


Key Recommendations

• fair registration practice code

• information for applicants

• criteria for decisions

• reasonable costs


• alternative evidence for unattainable documents

• training

• third party review

• self audits with reporting

• appeal procedures


2002 – ISO 17024

General requirements for bodies operating certification schemes

for persons


ANSI

SCC


2006 Fair Access to Regulated Professions Act


Key features

➪ access centre ➪ Fairness Commissioner ➪ audit


Consider self audit tool

• gap analysis

• demonstrate commitment

• continuous improvement

• achieve best practice


Focus on entry to practice


Conversion to standards38 – registration10 - organization


Examples• The College has policies and procedures

related to confidentiality and privacy

• The College has clear processes for reviewing, evaluating and making changes to registration requirements


Examples• The College has uniform competence

requirements

• The College ensures that individuals with a conflict of interest are not involved in the decision-making process related to an applicant.


Format

• standard definition

• evidence

• analysis

• recommendation

• ISO reference


Key Findings

• identified no major gaps or concerns

• links between College and third party hard to evaluate

• Identified new areas for policy work activity


• identified areas where evidence weak

• tedious

• how often necessary?


Key Outcomes

• confidence in fairness and transparency of process

• evidence of performance all included in one place

• celebration of excellence


Next steps

• develop work plan to implement recommendations

• share with government and other regulators


Speaker Contact Information

Jan Robinson

Registrar & CEO

College of Physiotherapists of Ontario

e-mail : jrobinson @collegept.org

www.collegept.org

Documents

Measuring Regulatory Excellence: Key Performance Indicators