Applying performance based funding to literacy and essential skills

  • Published on

  • View

  • Download

Embed Size (px)


Applying performance based funding to literacy and essential skills Boris Palameta, Karen Myers, Natalie Conte. January 16, 2013. The context. Transition to a new economy, in which skills are the new currency - PowerPoint PPT Presentation


Slide 1

Applying performance based funding to literacy and essential skills

Boris Palameta, Karen Myers, Natalie Conte

January 16, 20131Transition to a new economy, in which skills are the new currencyMany jurisdictions moving towards integrated service delivery models along an employment continuumInterest in re-aligning incentives to improve outcomes for job-seekers, employers, and tax payersCan performance-based funding (PBF) drive system-wide change?What can we learn from the experiences of other jurisdictions?Todays presentation draws a State of Knowledge Review that was conducted in partnership with Workplace Education Manitoba and funded by HRSDCs Office of Literacy and Essential Skills

The context22Literature review - Review evidence on various PBF models (Canada, US, UK, Australia) Expert review Interviews with PBF experts in other jurisdictions Consultations Consultations with practitioners and government officials in Manitoba and Nova ScotiaExpert panel Canadian Economics Association conference, June 2012

Throughout the process our approach was guided by input from the project reference group which was comprised of officials from MB & NS. Our approach33A tool for allocating resources to service providers based on measurable performance targetsShifts the focus from inputs to outcomes Assumption is this shift will drive innovation in service delivery and achieve desired long-term outcomes4Introduction What is performance based funding?4Design matters - PBF systems are complex and vary widely in design and effectivenessBetter design can mitigate risk PBF risks generating unintended consequences, but second-generation designs are more successful in mitigating these risksPromising approaches Establish meaningful links between practice and performance by paying for client progress along employment and learning pathways Intermediate outcome milestones (tipping points) as performance indicatorsState of knowledge Key findings55Examples: Work first Job placement; human capital development; poverty reduction; productivity SYSTEM GOALSPolicy objectivesTarget populationExamples: Employment status; income status; work readiness; human capital, demographic

Outcomes of interestImmediateShort-termLonger-termClient outcomesProcessPerformance indicators PROCUREMENT MODELLess competitive-non marketOpen competition-quasi-market

DESIGN OF INCENTIVE SYSTEM% service-based paymentsBy outcome By client characteristicsFinancial Non-Financial (e.g. star ratings)% outcome-based paymentsBenchmark attainment (x$ if x% of clients achieve outcome A)Payment per outcome (y$ per each client achieving outcome A)By local economic conditionsBy client characteristicsBy speed of placementPayment based on absolute performancePayment based on relative performance

Type of incentiveScale of riskPerformance targetsPayment weightingAdjustment for factors outside provider control Competition for incentiveState of knowledgePBF systems are complex and vary widely6U.S. Job Training Partnership Act (1982-1998)Australia Job Network / Job Services (1998-current)Washington St. Student Achievement Initiative (2007 current)ContextEmployment servicesEmployment servicesAdult education (college system)Policy objectivesROI in human capital developmentWork first; job placement, cost cuttingEducational attainment leading to labour market attachmentPerformance indicatorsPost-program (employment and earnings at 13 weeks)Post-program (13 & 26 week job placements)In-program continuum of learning outcomes, from basic skills to credential attainmentFunding model & scale of risk6% of state allocations for incentive awardsCompetitive bidding up to 50% for outcome payments2% funding cut reallocationPerformance paymentFor meeting federal or state-set targets (% attaining outcome)Per outcome achievedPer point gained within learning continuum (momentum points model)Leveling playing field Targets adjusted to local conditions and client characteristics Client tiering; payment weighted by type of clientMomentum points determined by principle of equivalent effort Competition between providersIn some statesYes; star ratingsEach college assessed against its own historical performanceThree contrasting models77Even small amounts of PBF may change behaviour, but not all changes are in the desired direction Early models particularly fraught with unintended consequences (cream-skimming, parking, gaming)Second generation models are more promising with built-in features that aim to avoid these pitfallsKey to mitigating risk is not only careful system design, but also commitment to continuous improvement

State of knowledge 2. Better design and monitoring can mitigate risk88Choice of measures crucial in determining incentive architecture Poorly chosen performance measures may create conflicting incentives obtaining performance payments vs. serving clientsPerformance measures have often been:1) Outside provider control; i.e. based entirely on outcomes that happen after clients leave the program No clear connection between services providers offer and outcomes they are paid for

2) Based on attainment of levels rather than gains from a starting pointIncentives to pick winners

Performance measures99Performance measures have often been:3) Poor proxies for qualityProgram outcomes of interest are often long-delayedPerformance measures typically use short-term proxies (e.g. employment at 13 weeks) for outcomes of interest (e.g. longer-term employment)But chain of evidence is often lackingNo clear connection between the short-term outcomes providers are paid for and longer-term program impactsHitting the target, missing the point

Performance measures (contd)10101) Use in-program performance measures > In-program measures establish a more immediate and meaningful connection between day-to-day practice and performanceAllow providers to track progress in a timely fashion, understand where and why learners succeed and where they falter, and design interventions to accelerate progress2) Measure gains not levels > Most measures have focused on levels attained by clients at the time performance is assessed.Need measures that include starting points and magnitudes of improvement to convey information about providers impact on learner achievement.

Guiding principles for designing PBF systems that work

11113) Measure what counts > Avoid mission narrowing by ensuring that performance measures recognize the full range of program objectivesPBF changes cost/benefit calculus, may encourage development of costly but innovative services; on the other hand, what you do not pay for may be left undone4) Identify key milestones > Identify intermediate milestones that can be used to track the progress of clients who may enter at different points (e.g. with different levels of skill, employment readiness, etc.)Select milestones based on points along the pathway where learners stall or struggle meaningful transitions

Guiding principles for designing PBF systems that work

12125) Monitor system performance > Build a continuous learning process to respond to unplanned behaviourE.g. teaching to the test

6) Right-size incentives > Ensure performance incentives are neither too big nor too smallToo big risk management rather than innovationToo small if costs of meeting performance targets > performance bonuses, incentives will be ignored

Guiding principles for designing PBF systems that work

13137) Flexible approach to performance targets > Pre-set performance targets are often either too ambitious or not ambitious enough; both can lead to strategic behaviourA more open-ended approach encourages continuous improvementE.g. awarding performance dollars according to momentum points (i.e. total number of milestones achieved along a learning pathway) 8) Ensure all targeted clients are served > Level the playing fieldDesign incentives using the principle of equivalent effort whereby each momentum point should require roughly the same intensity of effort to attain. Recognizes that clients with more barriers may require greater effort to transition between milestones.

Guiding principles for designing PBF systems that work

14149) Build provider capacity > Providers may lack knowledge or resources to respond effectively to incentivesLimit competition for performance dollars; encourage collaboration to build tools and practices10) Link in-program measures to post-program impacts > Use longitudinal research to establish a chain of evidence between intermediate milestones (potential tipping points) and longer-term, post-program impacts (e.g. employment, earnings, etc.)Follow-up with learners to establish the connection between measured performance and client success in the long-termUse results to refine and improve performance measurement framework

Guiding principles for designing PBF systems that work

1515Washington State Student Achievement Initiative16Key transitions milestones within a students pathway (identified by research as tipping points). Provides incentives to focus on full range of skill levels16Rewards achievement of key milestones Encourages client progress by rewarding achievement of key milestones that, if reached, are associated with further progress and ultimately long-term labour market successFocuses on balanced set of in-program measures Which helps providers understand where clients succeed and where they falter, and thus provide the data to drive innovation Driven by a balance of competition and collaboration - Allocates performance dollars according to total number of milestones achieved. Thus while providers have strong incentives to innovate, they are not in competition with each other. Indeed they may be motivated to collaborate to improve outcomesState of knowledge 3. Key features of promising approaches1717


View more >