Background - NYSILC · Web viewObjective # 1 – NYSILC Operations16 Objective # 2 – SSAN22 Objective # 4 – NYSCB IL FFS29 Objective # 5 – CBILCO31 Objective # 7 – CIL Coaching33

NYSILC 2016 SPIL Evaluation Report

Consultant’s Report on the Third Year of the 2014-2016

State Plan for Independent Living (SPIL)

Submitted by Alan Krieger

Krieger Solutions, LLC

May 5, 2017

ContentsBackground..................................................................................................................................1

2016 SPIL Evaluation Findings.......................................................................................................2

Summative Analysis – How well were the objectives and targets met?...................................................2Overview:______________________________________________________________________________2Summary of the evaluation of each of the objectives and consultant observations.___________________3Objective 1: NYSILC Operation -____________________________________________________________3Objective 2: Support a Statewide Systems Advocacy Network (SSAN)______________________________6Objective 4: NYS CB (formerly CBVH) will provide IL services on a fee-for-service basis to eligible

individuals who are legally blind.________________________________________________________8Objective 5: Capacity Building_____________________________________________________________9Objective 7: Coaching and consulting for Centers_____________________________________________11Objective 9: Develop a Database__________________________________________________________12

Overall Summary..................................................................................................................................13

Formative Analysis...............................................................................................................................13

Recommendations................................................................................................................................15

SPIL COMMITTEE REPORT...........................................................................................................16

NYSILC 2016 SPIL Objectives/Evaluation Summaries...................................................................16

Objective # 1 – NYSILC Operations........................................................................................................16

Objective # 2 – SSAN.............................................................................................................................22

Objective # 4 – NYSCB IL FFS.................................................................................................................29

Objective # 5 – CBILCO..........................................................................................................................31

Objective # 7 – CIL Coaching.................................................................................................................33

Objective # 9 – Database......................................................................................................................35

NYSILC SPIL Evaluation Report, FY 2015-2016

Background

This evaluation covers the activities from October 1, 2015 through September 30, 2016 related to objectives in the statewide plan for independent living. This is the final year of the plan and some objectives were already complete or otherwise not active this year. The evaluation consultant had limited discussions with most of the committees about their evaluation process. NYSILC staff, as always, did a great job pulling together the data and working with the committees to finalize their assessments. The evaluation consultant participated in committee meetings to review the data collected and to continue to focus on the shift from activity or inputs to outputs or impacts. The evaluation consultant did not review any of the original data, only what the Council staff pulled together.

As noted in previous reports, the Council has come a long way with writing a more specific and measurable plan. The objectives and targets are more outcome or impact oriented; vague terms and standards have been defined more clearly to make the evaluation more objective. The Council is currently in the process of developing a monitoring protocol so the Council can take an active role in measuring success in accomplishing the objectives and to get an early awareness if issues arise so they can be addressed during the program year rather than after the fact.

This year’s evaluation consulting activities included helping finalize the objectives and targets for the new SPIL, and continuing to look at “value added” components for each objective that focused more on outcomes and results. Earlier plans consisted mostly of activity or process oriented objectives. They discussed what the Council or its subcontractors would be doing. The new SPIL has a few more outcome oriented objectives, but in the push to get it finished, a number of activity oriented objectives remained in place. So, we continued to add some “value added” elements to bring more of an outcome orientation to the evaluation.

There were several targets that were not clearly defined in the SPIL (e.g. what is a “system change”) and committees have taken steps to better define some of these so they can be more accurately evaluated.

At the June 2015 Council meeting additional training was provided for Council members on the evaluation process in advance of the development of the new SPIL. The training focused on developing outcome based goals and objectives to help continue the transition from process/activity objectives to outcome objectives. In addition, as new individuals continue to be appointed to NYSILC, it may be useful to plan additional training for them. New members continue to be provided a link to the webinar recorded earlier. The outcome evaluation training materials developed and presented the first year the evaluation consultant worked with the Council are also included in the new member orientation manual.

- 1 -


Some objectives from the previous SPIL were purposely delayed to provide enough time to develop the RFP’s. The final one or two years of these objectives were extended into this SPIL with the use of unspent Part B funds.

Similar to the earlier years, each committee developed an evaluation process that generally relied on the NYSILC staff gathering the relevant data from records for projects being conducted by NYSILC, or from the DSU for projects that were handled through RFP’s. Some additional work was done to evaluate some of the “value added” components. Committees are paying more attention to the actual impact of the projects, not just the amount of effort that was expended.

Each of the committees conducted their evaluation between November 2016 and February 2017. The phone calls in which the SPIL evaluator participated usually have very few committee members participating. This low level of involvement is a concern. Only one committee had a SPIL evaluation session where they were fully engaged in discussion. This would be the goal for every committee. Written summaries of the committee meetings were transmitted to the SPIL evaluator who then drafted an evaluation report based on those summaries and committee discussions. The full committee summaries are appended to this report.

2016 SPIL Evaluation Findings

Summative Analysis – How well were the objectives and targets met?

Overview:

Most of the objectives in the SPIL initially look to be very straight forward to evaluate. While the objectives themselves are not usually written in measurable and specific terms, they each have measurable indicators that are measurable. These indicators generally relate to whether or not certain activities took place, how many people participated in the activity, what the impact was of the activities, and/or how often or to what degree the activity was accomplished. On a deeper review, some questions emerge. Some of the objectives were technically met, i.e. the number of activities/people served were achieved. However, the impact or outcome of these activities in some cases appears to be minimal and not what the objective’s intention was.

The reverse is also true, some of the targets that the committee felt were “not met” due to data problems or technicalities were partially met when the data was examined more closely and the intention or desired outcome was considered. In some cases, the objectives or indicators were written too specifically. Therefore, the evaluation must show that the objective was technically not met, although the intention appears to have been met. Similarly, by writing

- 2 -


some of the objectives too broadly, the objectives were technically met although the impact appears to be minimal and not what was hoped for.

This report is generally based on the technical rating – did the objective and indicators, as written literally get met. In addition, the consultant added comments where a different rating would actually be more accurate.

There were 10 objectives in the 2014-2016 SPIL. Four of the objectives (#6 Voter Registration, #7 CIL Coaching and Consulting, #8 Capacity Building and #10 Deaf-Blind SSP) were carried over from the prior SPIL for only one or two years, so were inactive this year. There was no IL Conference this year, so Objective 3 was also inactive. The 5 remaining objectives had 30 specific measurable performance targets set for this year, but only 28 were active. The Council has adopted a scale or range for rating each target and objective. The scale includes: exceeded, fully met (100%), substantially met (at least 60%), partially met, not met/no progress. This reflects the understanding that even when an objective or indicator is not fully met, if there was substantial progress, that has substantial impact and is worth noting.

Overall, based on strictly numbers, the Council fully met and exceeded the measures for one objective (#2). Three of the five active objectives were substantially met (objectives 1, 4, and 5) and one was partially met (#9). The fully completed rate of 20% is lower than any previous year. Four of the five (67%) were at least substantially met which is slightly lower than previous years, but overall, the SPIL was substantially met for the past year.

For the five active objectives, there was a total of 30 performance targets, but only 28 were active this year. Thirteen exceeded the target (46%) and five met the target (18%), so 18 out of 28 active targets were accomplished, a rate of 64%, about the same as last year, but falling short of the ambitious goal of 90%.

As noted in the introduction, these numbers do not tell the whole story. As explained in the report which follows, the numbers often show the extent of the activities that were conducted, but don’t always reflect the intention of the objective or the target in terms of impact or outcomes.

Summary of the evaluation of each of the objectives and consultant observations.

(For more detail see the SPIL committee’s full report, which follows the consultant’s report)

Objective 1: NYSILC Operation - In order to effectively coordinate, monitor, and evaluate the SPIL, NYSILC will strengthen its operations and capability.

This objective has nine targets, three were exceeded, three were met, one was

substantially met and two were partially met. Overall, the objective was substantially met.

As noted in prior year reports, this objective was revamped in this SPIL to show more of

- 3 -


an impact or outcome. In the earlier SPIL, one target was that meetings were held or the 704 report was completed. In this SPIL, some qualitative elements were added to these targets – Council meetings had to have a quorum so business could be transacted. The 704 report was completed accurately and submitted on time to RSA (which has since transitioned to the Administration for Community Living (ACL).

In actuality, the six targets that were exceeded or met have for the most part been met for many years and this does not show a “strengthening” but shows a maintenance of good or excellent performance. To measure “strengthening” the Council would need to identify areas of weakness, measure the baseline and show improvement after one or more years.

Two of the targets relate to Council administration (704 report and audit). These have been fully met for many years. One target related to Council operation – the number of meetings with a quorum – has also been met for many years. It was an issue several years ago, and did show a strengthening, but since it’s been met for several consecutive years, it no longer shows a strengthening, but a maintenance.

Two other targets related to Council operation (number of people actively serving on committees and number of issues addressed) could be a basis for showing strengthening.

Number of people actively serving on committees – the Council reports a very high number (45), but each time I have participated in committee meetings, the numbers have been extremely low. The definition of “active” participation may need to be clarified. I have difficulty accepting the Council’s report on this target based on my experience, but without more complete data, it should be counted as the committee identified it, as being exceeded. However, perhaps it is a matter of understanding how this number was counted toward the outcome. Does it reflect the members who just attend the committee meetings, or should it be adjusted down to just those individuals who actively participate during the meetings or to those who attend on a regular basis? The Council will be now be using Outlook invitations as a way to ensure good attendance at each committee meeting. They may wish to additionally develop a definition of what constitutes an “active” committee member.

The “number of issues addressed” was a vague term that the committee has worked on to clarify and now refers to the fact that the Council took some affirmative action in relation to the issue and in many cases, the Council has recorded the impact of those actions. This third target shows real improvement in Council operations and impact.

Two other targets relate to the overall accomplishment of SPIL objectives and targets. One was substantially met (#of targets achieved) and the other was only partially met (# of objectives achieved). The Committee discussing this felt that since this was out of the Council’s direct control, it should not be counted against the Council if the numbers fell short. This has been dropped from the upcoming SPIL. They also felt that the group of outcomes should be specific to the operations and activities relevant to the council. I disagree. The Council sets the

- 4 -


targets and while it does not select the contractors, the Council is allocating funds for these projects and is ultimately responsible for the administration of these funds. I think the Council should be accountable for this, perhaps in partnership with the DSU. I have indicated repeatedly over the years that the targets the Council has set (often in the 80-100% range) is unrealistically high, and the Council has never met these targets. Ambitious outcome measures are rarely met at such a high level. If the targets were set at 50% fully achieved and 20% substantially achieved, this would be a strong result and in many years, the Council could have met this target. I would encourage the Council to reinstate this as a measure of success in future years.

The final two targets relate to youth participation. One in the broader community and one in terms of participation on the Council and committees. The one for the broader community involves providing scholarships for youth to attend leadership development programs. While only 5 did so this year, the Council also helped support an event where 6 individuals from NY attended. Therefore, it seems to me that this target was substantially met, not partially met as the committee concluded. One great addition to this target is a follow up survey that the Council developed to assess the value of the experience. This helped move this target from an activity – youth attending an event; to an outcome/impact measure – youth increasing their leadership skills and active involvement in the community. I have recorded this as being “substantially met”.

The second youth target related to youth involvement with the Council. Five new members were appointed to the Youth Leadership Subcommittee and one was appointed to the Council. The committee is becoming very active including working on a strategic plan for young adults. Now that the Council has a full complement of young adult members and a strong youth subcommittee, it may make sense to shift this target to “maintaining strong representation of young adults on the Council (minimum of 4 representatives), and maintaining a strong and active youth leadership subcommittee (minimum of 6 members with an active agenda)”. For now, this can go in as “value added”.

`Summary:This objective had nine targets. The committee reported that three were exceeded,

three were met (67% met or exceeded), one was substantially met, and two were partially met. My assessment is that three were exceeded (although the number of active committee participants is not clear, based on my experience); three were met; two were substantially met and one was partially met. Overall, in either case, the objective was substantially met based on the number of targets met.

As the Committee reported their findings: the six of the nine targets that were fully met or exceeded include: (numbers in parentheses are # accomplished/ # of original target)

(Note: The first four bullets below are ways to show the Council is operating effectively administratively. They don’t show mission related impact, but do show competent administration.)

- 5 -


Number of full council meetings held during the year with a quorum (4/4):

Number of people actively serving on NYSILC committees (duplicated count): 66/45 (104 listed, but 66 actively serving). Active service is difficult to measure, but would be a meaningful measure. This target was not carried over into the next SPIL since as it currently stands, since the committee felt it is not a meaningful measure as is.

Financial audit completed “unqualified” and 990 forms filed fully, accurately, and on time: 100%.

704 Report completed with DSU to RSA (ACL) fully, accurately, and on time: 100%.

Number of issues addressed by NYSILC committees (23/12).

Number of young adults appointed to NYSILC or joining youth leadership subcommittee: (6/2). Five on youth subcommittee and 1 on Council.

The following were substantially or partially met:

Number of young adults participating in a training sponsorship: (5/12).

Percentage of SPIL performance targets that are met on an annual basis (64%/100%). In 2016, there were 5 active objectives with a total of 28 active targets related to them. 13 of these targets were exceeded, 5 targets were met, 2 targets were substantially met, 5 targets were partially met and 3 targets were not met. Totaled, that results in 18/28 targets met or 64%.

Percentage of SPIL objectives that are fully completed on an annual basis: 20%/100% (partially met). There were a total of five objectives identified by NY SPIL. One of those five objectives were fully accomplished; the Statewide Systems Advocacy Network (SSAN) exceeded all seven of its targets.

While much of the Council’s impact is measured in the accomplishment of the SPIL each year, including the nine targets for objective 1 and the other major objectives that follow, there are several additional impacts worth noting. These include: maximizing cooperative working relationships; developing, distributing and posting quarterly newsletters and an Annual Report; maintaining a website with timely information; developing mechanisms to solicit and recruit new council members; and conducting a statewide needs assessment. These types of outcomes should be reflected in any full evaluation of the Council.

- 6 -


Objective 2: Support a Statewide Systems Advocacy Network (SSAN)

New Yorkers with disabilities will be active and will organize to promote disability rights working with priorities communicated by NYSILC to address issues identified by the statewide needs assessment through the support of a Statewide Systems Advocacy Network (SSAN) and statewide coordinator.

There were seven outcomes related to this objective and all seven were exceeded. This objective therefore was fully met and exceeded. This objective has also been refined over the years to become more outcome oriented and it appears that this project has produced some strong outcomes. The most impactful outcome is the number of successful system changes the network worked on. While the changes can’t be tied specifically to the network’s efforts, the combined efforts of the network are substantial and it’s reasonable to assume that they had some impact on the system change efforts along with those of other advocates.

o Number of successful system changes (9; target was 2). Exceeded. A strong definition was developed for what constitutes a “systems change” and once again there were several successful system changes over the past year that the SSAN network was actively involved in. The definition includes that the SSAN coordinator will take a leadership role in the process and that SSAN providers would be substantially involved. Another element is that the change must be statewide or lead to statewide change. It appears that three changes the network worked on meet this enhanced definition, including getting new laws passed and securing funding for key initiatives. In addition, six unwanted changes were blocked in the Executive budget and legislative session. This target was exceeded with some significant impacts.

o Number of partnerships established (140/70). Exceeded. As noted last year, there is no standard or criteria for what constitutes a meaningful or substantial partnership. Four impressive examples are cited, but it’s not clear how many of the 140 new partnerships meet this degree of involvement. However, the SSAN reports on partnerships do include the activities the partners participate in. I have never received a copy of these reports. The Council may want to have the Monitoring Committee verify this information in the future. Having a clear and substantial standard for this would make this number more impressive and significant. Strengthening this standard might require a lower overall number as a target. Actively carrying out a number of activities would be a more meaningful measure.

o Number of education alerts (752/616) Exceeded.

o Number of public education activities (470/280) Exceeded.

o Number of grassroots organizing activities (197/84) Exceeded.

- 7 -


o Number of written/oral testimonies (144/84), Exceeded.

As noted in prior years, the number of education alerts, public education activities, testimonials, etc. is impressive. It does show that the network is active, engaged in local and state advocacy and education. The large numbers show that at least some of the partnerships are meaningful and taking on active roles. It would be very interesting to be able to show some impact in terms of getting more people involved or successful change of policies.

This is difficult to measure, but worth continuing to consider. With some email systems, it is possible to count how many of the alerts are “read” or “opened” and that would be a good place to start. If the alerts ask people to take action, it could also include a request to “reply to let us know” that the action was taken. This does ask the reader to do an additional step and requires the network to undertake additional record keeping, so instead of doing this across the board for all 906 alerts, perhaps it could be done with some of the more critical alerts as a small sample (5-10 per year) to get a sense of how effective this network is. With public education activities and grassroots organizing activities, it would be helpful to estimate the number of people in attendance. Even better would be to do an email follow up to those in attendance asking for a brief report on what they might have done as a follow up. In both cases of asking for a response, only a small percentage of those who actually do something will respond, so it’s important to keep the targets small and reasonable, but generating even a small response shows that there is a real impact from these activities.

o The number of centers without a SSAN contract who are involved in the network (3; target 2). Target exceeded.

This shows the importance of the funding for centers to take an active role in the network. While it seems easy to pull in many volunteer partners, it is not as easy to bring in Independent Living Centers. As part of strengthening the evaluation process, the committee developed a definition of what it means for a Center to “participate”. This has been defined as “participating in the action alerts and receiving training to work on the outcomes”. It may be worth exploring what the barriers are to Center involvement at this level and see if there is a way to engage more Centers in the network without additional funding. It would seem reasonable that this number increases each year, since once a Center is involved, it’s likely to stay involved and bringing in an additional Center each year might be a good target.

Objective 3: IL Conference - there was no conference held this year, so this objective was not active.

- 8 -


Objective 4: NYS CB (formerly CBVH) will provide IL services on a fee-for-service basis to eligible individuals who are legally blind.

For eligible individuals who are legally blind, increase improved access at home or in the community and/or independence in their own lives, by supporting NYSCB to provide IL Fee-For-Services (FFS) and increase the number of CILs or SCILs in the network providing NYSCB FFS.

There were five outcomes related to this objective. Two outcomes (or targets) were exceeded, one was met, one was substantially met and one outcome was not met. Overall, this objective was substantially met.

One target was substantially met:o Number of legally blind consumers receiving NYSCB Fee-For-Service (FFS):(305;

target was 350; 87%), very similar to last year. At this point, if this were to continue, it would be helpful to ascertain why the leveling off at just over 300 and if the 350 target was not realistic. However, this objective will not be included in the next SPIL.

One was met:o Securing two additional CILs or SCILs to provide NYSCB IL Fee for Services. Two

additional centers signed on, meeting the target. However, a follow up survey of involved centers showed that there was little actual engagement, so while this target was technically met, it fell short of the intention of building a network of providers.

Two targets were exceeded were:o Number of consumers experiencing improved access at home or in the community

(163; target - 100)o Number of consumers experiencing increased independence (163; target - 125).

The targets set were 100 (32% of people served) and 125 (40%) respectively. For both targets 163 people (53% of people served) achieved this, exceeding both the raw numbers and the percent of accomplishment. These seem to be the most substantive targets and since both were exceeded, I’d say the objective overall was achieved.

The final target was to ensure that at least 85% of legally blind people receiving the services and responding to the survey reported that they were satisfied with the services they received. Unfortunately, the survey was not successfully completed. As a result, this outcome was not met. However, if in fact 53% of the people served showed significant changes, it’s likely that at least that many were satisfied with the services received. So, while technically this target was not addressed, in reality, it’s safe to assume that its intention (consumer satisfaction) was at least partially and most likely substantially met.

- 9 -


Objective 5: Capacity Building

Improve the capacity of the IL network to address priority unserved and underserved populations from the statewide needs assessment.

There were a total of five targets for this objective, but only three were active this year. One was exceeded and two were partially met (but both over 50% putting them close to “substantially met). Overall, therefore, this objective was substantially met.

The SPIL called for eight funded projects to serve unserved/underserved populations. There were unexpended funds that allowed an increase so that ten centers were awarded contracts. As a result, the targets were increased proportionately.

o Number of people served (target 480; adjusted to 570; actual: 317 or 56%). Partially met. There was some question about how to count the “number of people served”. It was agreed that establishing a Consumer Service Record (CSR) and providing services to these individuals is what would be counted, not the number of services provided. On that basis, the project met 56% of the target. The data included a detailed list of the kinds and numbers of specific services provided. Providing a service, doesn’t mean it was done well or had the hoped-for impact or benefit. The committee had decided to conduct a follow up survey at the start of this year (fall 2016) to get a better sense of how these services helped consumers in these unserved/underserved populations. This is a good example of moving from measuring activity (services provided) to impact (benefits of services). This survey was not conducted in 2016 and has been rescheduled for 2017. It will be combined with an assessment of self-sustainment.

oDeveloping a network of community organizations (target 40; adjusted 50; actual 67). Exceeded

Part of this project was developing a network of community organizations to support the service delivery and/or assist with recruitment. Again, the question is what value does it represent to have an organization listed as a contact? The committee discussed trying to identify some of the benefits, but due to the limited size of the grants didn’t want to make this administratively cumbersome. One suggestion was for the grantee to indicate when each agency was contacted what kinds of services/support the agency was willing to offer. The Council is considering developing a checklist of these kinds of services to make it easier to collect and report the data. As it currently stands, we know that 67 organizations were contacted, technically exceeding the target, but telling us nothing about the value of these contacts. Since the number of consumers served is well below the target, it does not seem that these contacts provided the desired value. This concern was communicated to the Outreach Subcommittee, who was responsible for establishing the guidelines for the CBILCO “How To” manuals. They asked for additional information in the partnership section of the manual to help ascertain the quality of the relationship. This will help determine the value of these partnerships.

- 10 -


The ultimate goal is for these projects to become self-sustaining with consumers receiving services after the funding ends, and to develop and document best practices for other centers wishing to serve these populations. There were three targets related to this:

o Raise funds to support continued operation after the funding ends. (target: $66,666; adjusted $83,332; actual 43,297 has been raised; 52%) partially met. Some of this is “one time money” – a donation or grant which may not help with long term sustainability. Other funding is multi-year or fee for services which have a better chance of moving into self-sustaining services. Also, this is a capacity building project and if the capacity to provide these services is fully developed, some of that can be integrated into the on-going services of a center, supported by the network developed, which could allow for some self-sustainment without the same amount of funding. As the committee noted, the real test of this target will be one year after the funding ends. For now, this target was partially met.

o Have several these programs become self-sustaining. None were expected to achieve that so early in the program, so that target is not applicable for this year. The Committee considered this “met” – the goal was zero and we achieved zero. I consider this “n/a” for this year and excluded it from the overall calculations.

Production of “how to” technical assistance manuals to help other Centers develop services for these unserved/underserved populations. Again, this was planned for at the end of the project, so is also not applicable at this stage. The Outreach Subcommittee sent out an email to the CBILCO projects in October 2016 to inform them about the guidelines and protocols related to the “How To” manuals. A progress conference call took place in April 2017. Encouragingly, several centers reported that they have already begun to develop these manuals. The Committee again considered this “met” – the goal was zero and we achieved zero. I consider this “n/a” and excluded it from the overall calculations.

Objective 6: Voting Rights Network - This objective was only funded for the first two years of the SPIL. It was not active this year.

Objective 7: Coaching and consulting for Centers his objective was only funded for the first two years of the SPIL. It was not active this year.

Objective 8: Capacity Building – Carry Over Not active in 2016

Objective 9: Develop a Database

- 11 -


Designate funds for the purposes of developing technical specifications and establishing a database to compile, analyze and interpret data from the statewide network.

There were four targets related to this objective. One outcome was met, one partially met and two were not met. These targets are difficult to understand and measure. They use a percent to show completion of activities where in some cases there is no framework to measure percent of completion.

o Percent completion of database design (100%; target - 100%). Met.The committee selected a vendor and agreed upon a design, so this target was met.

o Number of centers contributing to the design (9%, target - 60%). Partially met.Four centers actively participated in discussions and online feedback, which contributed to the database’s design and development. The target was unclear, but the committee interpreted it as 60% of the 44 centers, or 26 centers. Four centers constitute 9% of the network of 44. The committee has indicated that for successful implementation it must do outreach to engage the majority of centers going forward.

The last two targets were contingent on the data base being ready for testing and reporting. Since it’s not at that stage yet, neither target was met. Overall this objective was partially met.

Objective 10: Support Service Provider (SSP) Not active in 2016

Overall Summary

Overall the objectives had very strong results even if only one was fully met. The committees continue to do a good job of clarifying and defining the terms used for evaluation purposes. As noted above, there are additional areas still in need of definition and specificity. On the other hand, some of the targets had unrealistically high targets (see objective 1 – SPIL outcomes).

When setting targets, it would be helpful to think carefully about the intention of the objective and what are some meaningful and measurable indicators that could show (indicate) that progress is being made for the objective. Objectives and targets should indicate progress towards a desired change which is stated in the goal statements. The Council has come a long way from the start in developing outcome based objectives and indicators. Continued attention to this will create better and better plans and evaluations that measure meaningful results and impacts.

Formative Analysis

- 12 -


The above analysis of the results of each objective focuses on “summative” evaluation, or evaluation strictly of numerical results. Formative evaluation looks at using the data to inform and improve the operation of the projects. In the current state plan, the Council made an effort to shift from objectives that were more activity based (measuring numbers served, numbers of events) to those that are more impact or outcome based (how the situation was overall improved). Some targets in the old plan were overly specific and detailed and were changed in the new plan to focus more on outcomes rather than specific outputs or activities – they indicate how things might change as a result of the plan’s efforts, not just what was done or what was produced.

Several the objectives show a clear shift in the direction of impact based performance targets and/or outcomes. This is a very challenging shift to make and the Council should be applauded for the progress made. From initially having no objectives, to in an earlier SPIL having 14 objectives, each with multiple performance targets that were mostly activity oriented, to now having a smaller number of objectives that are more outcome focused, is a large step forward. In the next SPIL the Council has taken this further, further prioritizing the objectives to a smaller number. This reflects a better sense of the Council’s role in the SPIL, an effort to maximize impact in light of the reduced funding due to the sequester. Continuing to include “value added” elements that focus on outcomes is a good strategy for continuing this development to more outcome based evaluation.

Developing a SPIL is a very large and challenging undertaking. The process rightly focuses primarily on how resources should be allocated. The time for writing objectives and performance targets is at the end when there is little time for reflection and revision. Committees may wish to begin the evaluation process on the first day of the plan implementation by looking at each objective and performance target and discussing upon what they would base their evaluation. If it’s not clear in the objective itself, the committee may want to develop a more specific set of targets and negotiate these with the contractors involved, so everyone is clear what will be evaluated right from the start.

The Council’s evaluation process focuses more on assessing “yes” or “no,” was the objective met or not? It could be helpful to also look at “why” an objective was met, exceeded or fell short. Was failure due to poor planning, poor implementation, or unexpected factors, or influences outside the control of those running that program? What was the reasons for success? Were the targets too low, were the providers exceptionally effective, or were additional resources made available? Whatever the reasons, the Council should look at what can be done in the following year to address these issues and continue to build on successes.

As these questions are answered, successes and failures can be analyzed and future plans can have more accurate targets. This can include revising unrealistic targets or objectives down, or increasing those that were understated. It can also include looking to reallocate resources or change program designs based on results of the prior year.

- 13 -


In several of these cases, adding quality criteria to the target (e.g. action alerts that are responded to), will bring down the number of “successful” outcomes, but will show outcomes that have a measurable impact, not just an activity.

The Council’s plans to have an increased role in monitoring project implementation will help answer some of these questions.

Value added: a number of objectives had “value added” components. In some cases, these components measured additional outcome results, and in some cases, they dug deeper to measure impact oriented outcomes. As these questions were answered, more clarity could be seen as to the true impact of the programs and initiatives. This is the primary area for continued growth for the Council in terms of evaluation: how to evaluate whether the programs and initiatives they support are making a difference in furtherance of the mission.

Learning from the prior SPIL, the Council has built in more development time for some of the objectives. This may need to be expanded to all the objectives. All new initiatives need time not only for the RFP process, but for establishing the program, developing tools and procedures, building networks, etc. One strategy discussed earlier is to consider year 1 of the plan a startup phase and then extend the project into the following plan to allow it time to gear up and operate for 3 years.

The Chairs of each of the committees serve on the Executive Committee. Therefore, it was decided last year to use the Executive Committee to help evaluate the evaluation process overall and the work of the evaluation consultant. That was not done this year. However, this will change next year as the council transitions to the new Monitoring and Evaluation Committee.

- 14 -


Recommendations

As the NYSILC committees continue to analyze and evaluate each of the objectives, they can continue to assess whether the targets are properly set, resources appropriately allocated, and systems effectively in place to ensure the best use of funds and the greatest impact on plan goals. Continuing to work to increase targets that focus on the impact of these objectives will help with this process, as will conducting conversations and/or surveys with providers and consumers to gather more detailed data in areas where performance fell short.

In some cases, where an objective or target was “substantially met” they may want to consider whether the target was overly optimistic and needs to be scaled back or if there are strategies for improving outcomes in future years.

ATTACHMENTS FOLLOW: SPIL Committee report: detailed evaluations of each objective

- 15 -


SPIL COMMITTEE REPORT

NYSILC 2016 SPIL Objectives/Evaluation Summaries

Objective # 1 – NYSILC Operations Executive Committee

Objective 1: In order to effectively coordinate, monitor, and evaluate the SPIL, NYSILC will strengthen its operations and capability.

There were nine outcomes related to this objective. Preliminary results indicate that NYSILC exceeded three outcomes (or targets), met three, substantially met (60%) one, and partially met two.

2016 Performance Targets:

Number of full council meetings held during the year with a quorum (Target: 4) : NYSILC successfully held all four full council meetings with a quorum during the year at the Troy Hilton Garden Inn on the following dates: November 13, 2015, April 15, 2016, June 17, 2016, and September 9, 2016. Outcome met.

o This is verified by the summary of minutes approved by the full council for each meeting. These summary minutes will be archived on the NYSILC website with audio clips for the meetings: http://www.nysilc.org/index.php/full-council-minutes-audio-and-minutes

o Value added:

o Evaluation: Exceeded, met, substantially met (60%+), partially met, not met, no progress, dropped, inactive.

o Comments: Target met.

Number of people actively serving on NYSILC committees - duplicated count (Target: 45): Based on the NYSILC 2016 Organizational and Committee document/listing, the following active NYSILC committees had the corresponding number of people on their committees during the past year (duplicated count): Executive Committee (7), Public Policy (13), Finance Committee (5), Recruitment (6), State Plan (SPIL) Committee (13), Emergency Preparedness (16), Youth Leadership Subcommittee (7), Development Subcommittee (9), Consumer Satisfaction Survey Subcommittee (9), Outreach Subcommittee (10), and Database Work Group (9). Total listed on committees: 104. Total actively serving: 66. Outcome exceeded.

o This is verified by the NYSILC 2016 Organizational and Committee Materials document/listing.

- 16 -

http://www.nysilc.org/index.php/full-council-minutes-audio-and-minutes

http://www.nysilc.org/index.php/full-council-minutes-audio-and-minutes


o Value added:


o Comments: Target exceeded. This is an example of how a target was established and “value-added” input was received to clarify how it could be improved. Rather than just count the number of NYSILC committee members on each committee (this is instead the starting point), the list is reviewed to determine how many of those individuals are actually “actively serving” on the committees. This number is used.

Number of issues addressed by NYSILC committees (Target: 12) : Based on the information from the July-September 2016 NYSILC quarterly contract report (which also reflects the end of the contract year), twenty-three issues where addressed by NYSILC committees during the past year. Total issues addressed by committees for the year: 23. Outcome exceeded.

o Some of the issues addressed/discussed by committees during the past quarter include: For the Executive Committee: the budget modification and

spenddown plan, restitution from the former employee, the NYSILC 3-year contract, recruitment of new members, SPIL submittal, the LDCEP program, consumer satisfaction survey, and database project.

For the Outreach Subcommittee: the communicate needed to be sent to the ten CBILCO projects as they enter the last year and coordinating an ad hoc group to determine the general guidelines for the “How To” manuals for the CBILCO manuals.

For the Public Policy Committee: review of the 2016 Presidential post-election survey.

For the Consumer Satisfaction Survey Subcommittee: review of all the updates to the process for the upcoming statewide CIL consumer satisfaction survey process.

For the Database Work Group: review of options provided by the Center for Health and Social Research (CHSR).

For the Emergency Preparedness Subcommittee: an update on Dave Whalen’s DDPC grant and Core Advisory Group (CAG) meetings.

For the Recruitment Committee: review and recommendations of new council members to fill vacancies for 2017.

This target also includes two sub-targets:

Issues successfully address by committees (Target 4): The council successfully address nine issues during the year. Total issues successfully

- 17 -


addressed by committees for the year: 9. Outcome exceeded. A few of the issues included: The Public Policy Committee completed the NYSILC 3-year Public Policy

agenda. It is posted online on the NYSILC home page at: http://www.nysilc.org/images/NYSILC_Three_Year_Public_Policy_Agenda.doc.

Committees and council with consultant complete NYSILC 2015 NY SPIL Evaluation Report: http://www.nysilc.org/images/Final_SPIL_Evaluation_Report_2015.docx.

The Outreach Subcommittee, with the assistance of the NYSILC office, completed an Excel spreadsheet, which will be used to monitor the quarterly and annual outcomes and results of the ten CBILCO projects. It will serve as a model monitoring tool.

SPIL Committee efforts end up facilitating an approved SPIL for 2017-2019:

http://www.nysilc.org/images/NY2017- 2019SPILApprovalLetter.pdf.

http://www.nysilc.org/images/ Clean_Copy_Draft_NY_SPIL_fifth.doc.

The NYSILC Needs Assessment Committee (NAC) reviewed the final draft provided by a consultant that resulted in the 2015 NYSILC Statewide Needs Assessment: http://nysilc.org/images/NeedsAssesFinalRep.docx.

Unresolved Issues (Target 4): Many of the unresolved issues identified during the year were eventually resolved. Only two unresolved issues were carried forward to the next contract year: the database project (SPIL objective # 9) and the 3-year NYSILC contract and related MWBE issue. During the year, five unresolved issues were resolved. Total unresolved issues addressed by committees during the year: 5. Outcome exceeded.

o Value added:


o Comments: Target exceeded. See recommendations for next (this) year for comments.

Annual financial audit completed “unqualified” and 990 forms filed fully, accurately, and on time (Target: 100%). Based on the information from the January-March 2016 NYSILC quarterly contract report, the NYSILC independent fiscal audit was successfully conducted and presented to the Finance Committee on 3/4/16. CPA Tom Hosey of Lubbe and Hosey reviewed the statement on the call with members and answered questions. He noted that NYSILC had an “unqualified” audit, which is the desired outcome. The Finance Committee

- 18 -

http://nysilc.org/images/NeedsAssesFinalRep.docx

http://www.nysilc.org/images/Clean_Copy_Draft_NY_SPIL_fifth.doc

http://www.nysilc.org/images/Clean_Copy_Draft_NY_SPIL_fifth.doc

http://www.nysilc.org/images/NY2017-2019SPILApprovalLetter.pdf

http://www.nysilc.org/images/NY2017-2019SPILApprovalLetter.pdf

http://www.nysilc.org/images/Final_SPIL_Evaluation_Report_2015.docx

http://www.nysilc.org/images/NYSILC_Three_Year_Public_Policy_Agenda.doc

http://www.nysilc.org/images/NYSILC_Three_Year_Public_Policy_Agenda.doc


formally accepted the audit and presented it at the full council meeting on April 15, 2016. The NYSILC 2014 990 was posted online at the following link: http://www.nysilc.org/images/NYSILC_Filing.pdf. Outcome met.

o This is verified by NYSILC quarterly contract reports which were shared in the full council meeting packets.

o Value added:



Annual 704 Report completed with DSU to RSA fully, accurately, and on time (Target: 100%). Based on the information from the January-March 2016 NYSILC quarterly contract report, NYSILC worked with its partners to help complete the annual 704 report. The NYS 2015 704 Report is posted online at the following link: http://www.nysilc.org/images/FINAL_SevenFour.doc. Outcome met.

o This is verified by the 704 Report posted on the NYSILC website.

o Value added:



Percentage of SPIL objectives that are fully completed on an annual basis (Target: 100%). In 2016, the NY SPIL had five active objectives. Based on NYSILC new evaluation criteria for its objectives (exceeded, met, substantially met (60%), partially met, not met – with the option of inactive or discontinued), this is an assessment of how many objectives were fully completed (i.e., either the objective reaching full completion or having all of their active outcomes exceeded or met during the year). Objective # 2 Statewide Systems Advocacy Network (SSAN) exceeded all seven of its targets. Overall, 1 of 5 objectives were completed or 20%. This is a 13% decrease from the previous year. However, objective # 1 NYSILC (this objective) was very close to meeting the criteria to being fully completed. Targets 6 and 7 represent the two measures that pose the greatest challenge for the council. They were discussed last year and were subsequently addressed going into the next plan so that it is expressed in a way that is realistic and reflects the council’s responsibilities. Percentage of SPIL objectives that are fully completed on an annual basis: 13%. Outcome partially met.

o This is verified by a variety of documents available to NYSILC and mentioned in this report.

- 19 -

http://www.nysilc.org/images/FINAL_SevenFour.doc

http://www.nysilc.org/images/NYSILC_Filing.pdf


o Value added:


o Comments: Target partially met.

Percentage of SPIL performance targets that are met on an annual basis (Target: 100%). In 2016, the NY SPIL had five active objectives with a total of 30 outcomes or targets related to them. NYSILC developed new evaluation criteria for its targets. It includes exceeded, met, substantially met (60%), partially met, not met – with the option of inactive or discontinued. Targets were evaluated according to these new criteria. Since the number of active objectives have decreased almost in half, it doesn’t make sense to compare them to last year (they will be less for each number by comparison). Based on preliminary information, 13 targets were exceeded, 7 targets were met, 2 targets were substantially met (60%), 5 targets were partially met, and 3 targets were not met (increase from last year). When the first three are combined (exceeded, met, and substantially met) it amounts to 20 targets with significant results. So, 22/30 or 73% which represents. This is a very good result and represents a slight decrease (74%) from last year. Outcome substantially met: (73%).

o This is verified by a variety of documents available to NYSILC and mentioned in this report.

o Value added:


o Comments: Target substantially met.

Number of young adults participating in a training sponsorship (Target: 12) . Based on the information from the July-September 2016 NYSILC quarterly contract report (which also reflects the end of the contract year), five applications were received for Pat Figueroa sponsorships. This reflects a decrease in activity compared to previous years. All five applications were approved. Four of the individuals attended the NCIL conference during the last week of July in Washington, DC. Two individuals went for the entire event and accompanied Zach Garafalo and Lyndsi Wickert who facilitate the NYSILC Youth Leadership Subcommittee. Two additional peers attended the one-day NCIL rally and participated in visits. They were supported by Aaron Baier and others from ATI Cortland. The newly elected youth representative to the NCIL board voiced her need to obtain support to attend future board meetings. She will get reimbursement to attend future NCIL board meetings. In addition, the

- 20 -


council supported the ADAPT youth summit this year which was held in Rochester instead of Chicago. Twenty youth from across the country attended the youth/young adult leadership training. NYSILC saw this as a unique opportunity to support an event so that several youth/young adults could receive leadership training that was taking place in New York. As a result, only five individuals attended trainings this year. In addition, six individuals from New York attended the ADAPT summit. NYSILC helped to sponsor the event. This represents 42% of the target. Total youth and young adult sponsorships of: 5. Objective partially met.


NYSILC created a survey monkey link to consolidate the results for 2013 to 2016 for Pat Figueroa Sponsorship recipients. This allows for a review of the collective responses to date. Most respondents felt that the trainers were knowledgeable about their topics (85% yes, 15% N/A), the training was useful to them (85% yes, 15% N/A), and the materials were useful (85% yes, 15% N/A). When asked if the training materials would help them to increase their leadership development skills, 92% replied yes and 8% said no. When asked if accessible accommodations and or formats were provided, 69% said yes, 8% no, 8% not sure, and 15% N/A. When asked for comments about their experiences, 2016 comments included:

o I had an amazing time getting the chance to advocate for people with disabilities and help stand up for the rights that they deserve. I also had fun getting the chance to tour the Capitol.

o Seeing so many people with disabilities gather together all at once was very emotionally moving for me as it showed how powerful working together can be and how so many people care about the cause of independent living.

o Being on the NCIL board will be a rich and rewarding experience.

When asked how the event/training changed their outlook on youth/young adults with disabilities,

2016 comments included:o I was surprised at how many youths independently attended the

event and stood up for their rights despite the social pressure from the many people around them who do not understand.

o That the differences of youth with disabilities do not make them less capable, and they will always have a bright future, no matter what obstacles they go through.

o As the first youth-at-large on the NCIL board, I have been empowered to continue to amplify the voices of youth with disabilities.

When asked are you willing to participate in activities in your community, sharing what you learned, and learning more yourself: (60%) were willing to

- 21 -


join a youth advocacy group in your area, (60%) were willing to volunteer at their local ILC, (60%) were willing to join or start a youth peer support group, and (50%) were willing to present on disability issues at an event in their local community.

At the September 16, 2016 full council meeting, the council held a youth forum. It was done to help evaluate the program and hear directly from Pat Figueroa recipients. In the end, it was felt that the Pat Figueroa sponsorships needed to be better promoted and more consideration should be given to applications with greater financial need or to events that have a significant cost to attend. The council will discuss ideas to better promote and expand the program.

o Value added:


o Comments: Target partially met.

Number of young adults appointed to NYSILC or joining youth leadership subcommittee (Target: 2). The NYSILC Youth Leadership Subcommittee became active during the past year. Zach Garafalo and Lyndsi Wickert facilitated the group. There are nine members active on the committee. They worked on a strategic plan for the committee and four members attended the NCIL conference. Five new members have been added to, and are participating in, the committee. Daniel Fuster was recommended to the council and will be appointed. This will add one youth/young adult member to the council. Total youth: 6 (5 youth subcommittee and 1 NYSILC). Outcome exceeded.


o Value added:


o Comments: Target exceeded.

o Recommendations for Next Year:

Before the recommendations are provided, the Executive Director should have framed the 2016 SPIL evaluation before engaging in it. First, this is the 2016 evaluation, so it focusses in on the last (third) year of the previous state plan. The objectives and the outcomes were discussed and improvements

- 22 -


were made going into the new state plan. Are they perfect? No, but they are improved based on input from the evaluation process we have.

As a result, after a lengthy discussion, it was recommended that the council utilize the presentation segments of its full council meetings in 2017 to review the current SPIL objectives and outcomes and look to improve the language – especially the outcomes. At a minimum, it would develop the “value-added” language to improve what should be in the SPIL. The SILC should seriously decide whether it should amend an active state plan and everything that is entailed.

Objective # 2 – SSAN Public Policy Committee

Objective 2: New Yorkers with disabilities will be active and will organize to promote disability rights working with priorities communicated by NYSILC to address issues identified by the statewide needs assessment through the support of a Statewide Systems Advocacy Network (SSAN) and statewide coordinator.

There were seven outcomes related to this objective. Preliminary results indicate that all seven outcomes (or targets) were exceeded.


Number of SSAN significant statewide system changes (Target: 2) . The SSAN participated in the following significant systems changes were achieved during the past year:

o Community First Choice Option (CFCO) Approved: On October 23, 2015, the State Plan Amendment was approved at long last, retroactive to July 1, 2015. This means NYS will finally begin implementing CFCO. It also means that the State is eligible for an additional 6% FMAP on CFC services, which will result in a significant amount of savings to the State. It should be noted that CMS approved the SPA without the amendment to the Nurse Practice Act to allow for Advanced Home Health Aides (AHHAs) because they consider this issue a state matter. However, the State still needs to amend the Nurse Practice Act to allow for AHHAs for full and robust CFC implementation. The legislature did pass AHHA legislation (see below for more detail) which we expect Governor Cuomo to sign. NYAIL and the SSAN have advocated for CFCO implementation for several years. Having CFCO means many additional long term services and supports are now available as State Plan services through Medicaid. The State has also committed to allocating savings from CFCO implementation to funding Olmstead implementation. NYAIL and the SSAN will continue to advocate for AHHAs as well as how the State should allocate savings realized through CFCO.

- 23 -


o NY ABLE Approved: Throughout the 2015 legislative session, NYAIL advocated for the New York achieving a better life experience (NY ABLE) savings account act, A.7767B (Gunther) and S.4472D (Carlucci). The ABLE Act will allow qualified New Yorkers with disabilities and their families to create tax deferred savings accounts to cover disability related expenses. The main benefit of these accounts is these assets will not count against programs such as Medicaid and SSI. Governor Cuomo signed this legislation into law on December 22, 2015. While this bill was not on NYAIL’s DPA, NYAIL met with both sponsors to ensure the best version possible and to ensure its passage. Many advocates within the SSAN also met with their legislators in support of the bill.

o Mental Health Education Bill Approved: The Mental Health Education bill, A.3887B (Nolan) and S.6046 (Marcellino), will ensure that mental health is included in any health education students receive. Through NYAIL’s Inclusive Education committee, the SSAN supported this bill throughout the 2016 legislative session by meeting with legislators, building coalitions of support, and responding to the action alerts NYAIL distributed in support. This bill was signed into law on October 1. While we acknowledge this was technically signed after the end of the reporting period, it was sent to Governor Cuomo, and all of our advocacy took place, within the reporting period.

In addition, the following systems change actions prevented negative activity from occurring:

o Successfully delayed the transition of NHTD and TBI Waivers into managed care: The Nursing Home Transition and Diversion (NHTD) and Traumatic Brain Injury (TBI) Medicaid Waivers were scheduled to be transitioned this year into managed care. Last year, advocates were successful in securing a one-year delay and a workgroup process to provide DOH with input to ensure a smooth transition. However, despite many workgroup meetings between DOH and stakeholders, there were many outstanding issues which had yet to be addressed. Stakeholders felt strongly that the draft transition plan did not protect services for participants in the two Waivers. Throughout the Budget process, NYAIL and SSAN advocates advocated for a delay to the waivers. As a result of this advocacy, the Budget delayed the transition to 2017, allowing additional time for DOH and stakeholders to address outstanding concerns. Throughout the year since, NYAIL and several SSAN centers have participated in those stakeholder workgroup meetings and provided written comments on DOH’s draft NHTD/TBI Waiver Transition Plan.

o Spousal and parental refusal protections maintained: The proposed Executive budget called for a change to the Medicaid regulation which allows a parent or spouse of an individual with a significant disability to refuse to support them financially. This

- 24 -


regulation allows the individual to keep their Medicaid eligibility without forcing the parent or spouse to give up all of their income. In the proposed Executive budget, the requirement would be for the parent or spouse to live separately in order to refuse to financially support the individual with the disability. This harmful proposal would force couples to divorce, or for the individual with the disability to go into an institutional setting in order to receive the necessary services provided under Medicaid. NYAIL strongly opposed this harmful proposal on our budget DPA and SSAN staff advocated for the legislature to reject this proposal in their one house budgets. Ultimately this harmful proposal was not included in the final budget.

o Prescriber Prevails maintained in final budget: The Executive budget proposed eliminating the crucial prescriber prevails provisions in Medicaid. This proposal would have repealed an important patient protection in the Medicaid program which restored “prescriber prevails” for several classes of drugs in the fee for service and managed care programs. A prescriber, with clinical expertise and knowledge of his or her individual patient, should be able to override a preferred drug. Different individuals may have very different responses to different drugs in the same class. Sometimes only a drug is effective or alternative drugs may have unacceptable side effects. Prescribers are in the best position to make decisions about what drug therapies are best for their patients. NYAIL strongly opposed this proposal in our budget DPA and SSAN staff advocated against it in meetings with legislators on NYAIL’s legislative day and throughout the budget process. Prescriber prevails protections were included in the final budget.

o Community Spouse Resource Limit Maintained: The Executive Budget proposal included a provision to lower the Community Spouse Resource Allowance to merely $23,844, the minimum allowed by federal law. Currently, New York allows a community spouse to keep the maximum allowed by federal law. As New Yorkers face some of the highest costs of living in the country, this makes sense. Lowering the resource limit to the lowest allowed by Federal law would have dire consequences for the community spouse, who is not on Medicaid and who may require assets in order to pay his or her own medical expenses. NYAIL opposed this proposal in our Budget DPA and SSAN staff vigorously opposed the proposal on NYAIL’s Legislative Day and throughout the Budget session. This proposal was not included in the final budget.

o Assisted Suicide bills die in legislature: Throughout the legislative session, NYAIL and our SSAN providers have actively worked to educate the legislature on the particular dangers people with significant disabilities face when assisted suicide is legal. At the beginning of the legislative session, NYAIL facilitated a webinar on the issue to help educate SSAN providers and others in the network to prepare them to speak to these particular concerns and dangers. NYAIL also helped to

- 25 -


organize meetings with legislators on the Assembly and Senate Health committees as well as an advocacy day in opposition. Many legislators who were undecided at the beginning of session decided to oppose the bills based on the education they received from the disability community. While this proposal will no doubt be raised again in future sessions, the disability community has made a real impact in the minds of many legislators.

o Uber bills die in Assembly: NYAIL’s transportation committee has been opposing any bills seeking to allow transportation network companies (TNCs), such as Uber and Lyft, to operate in New York unless they have accessible vehicles. NYAIL and many SSAN centers participated in multiple lobby days in Albany opposing the State allowing these services to operate unless they include accessibility. As a direct result of these meetings, Assembly members Weprin and Fahy asked for suggested language for a bill to include a percent of their fleet be accessible. Assemblyman Weprin did introduce a bill to mandate a percentage of these vehicles be accessible. While the Weprin bill did not move in the Assembly, neither did the insurance bill seeking to allow TNCs to operate in New York.

o Total number of significant system changes: 9 . Outcome exceeded. This is verified by the 2016 SSAN Yearend Report.

o Summative Evaluation: Exceeded, met, substantially met (60%+), partially met, not met, no progress, dropped, inactive.

o Comments:

Number of local partnerships and coalitions established by the SSAN network (Target: 70). The SSAN centers established 140 new partnerships and coalitions, exceeding the goal by 70 – a decrease from last year but still far exceeding the annual target. A few examples of the partnerships and coalitions include:

o AIM worked with a local restaurant to create menus in large print and Braille.

o ILCHV has been working with Transport Troy and many other local Troy organizations to put on an event called: The Collar City Ramble. The purpose of the Collar City Ramble is to shine a light on Complete Street proposals and examine the accessibility of trails along with bike roots, boating and other activities.

o FLIC re-established connections with the Tompkins County Board of Elections, as part of their voter registration efforts, and to get an article published about the Ballot Marking Device in the Ithaca Journal.

o WNYIL has started working with the New York State School for the Blind in Batavia.

- 26 -


o Total local partnerships and coalitions: 170 . Outcome exceeded. This is verified by the 2016 SSAN Yearend Report.


o Comments:

Number of educational alerts disseminated to local volunteers by the SSAN network (Target: 616). Distribution of alerts is one of the areas in which SSAN centers are most active with a total of 752 alerts distributed to local volunteers. This was a decrease compared to last year, but again exceeding the annual target. Subsequently, SSAN advocates collectively disseminated 156 alerts to their local volunteers. Examples include:

o Support HR 5689, the Disability Integration Act!o ADA Alert! Fight serious attack on the ADA!o Tell Your Senators to Support S.2680 and Reject Harmful Language in

Mental Health Bills!o Urge Your Senators to Support Passage of S. 2680, The Mental Health

Reform Act of 2016!o Protect CDPA - Stabilize wages and reimbursement!o Call Today in Support of the Mental Health Education Bill! o Get DIA into the Presidential Debate!

o Total educational alerts disseminated: 752 . Outcome exceeded. This is verified by the 2016 SSAN Yearend Report.


o Comments:

Number of local public education activities engaged in by the SSAN network (Target: 280). The SSAN Centers exceeded this goal by engaging in 470 public education activities in the past year. This exceeded last year’s activity. Examples of local public education include:

o WDOMI hosted a meeting with Yonkers City Council members to discuss their proposal for the City of Yonkers to amend its Affordable Housing Ordinance to set aside 10% of housing units which have been designated as affordable housing in any new or renovated housing developments for individuals living with mobility and sensory disabilities.

- 27 -


o RCAL provided local Disability Sensitivity presentation to New York Connects and OFA employees at Ulster County Business Resource Center (BRC).

o AIM met with a business council to discuss accessibility concerns in the district. The council agreed to conduct outreach to businesses and to work with businesses to address the accessibility issues, including fixing broken door openers. Also, an article about the importance of accessibility was included in the council’s newsletter.

o The Systems Advocate from Independent Living, Inc. attended a workshop on Americans with Disabilities Act Transition Plans and Other General Requirements provided by NYS Department of Transportation (DOT) and the Federal High Way Administration. The Systems Advocate presented on the role that ILCs can play in transition planning.

o Total public education activities engaged: 470 . Outcome exceeded. This is verified by the 2016 SSAN Yearend Report.


o Comments:

Number of grassroots organizing activities engaged in by the SSAN network (Target: 84). SSAN sites reported a total of 197 grassroots organizing activities for the year. This exceeded last year’s activity. Examples of grassroots organizing activities include:

o ATI organized local advocates to attend and participate in the NCIL March, Rally, and Capitol Hill Visits in Washington, D.C.

o STIC met with the Education Advocacy Team at STIC (parents, teachers, service coordinators, advocates, behavior specialists, professors) to review NYSED’s proposal for Superintendent’s Determination (affects high school students with disabilities). NYSED also requested public comments for numerous mandate relief proposals. EAT met to consider these proposals, too.

o WNYIL held an ADA Education Event at Delaware Park in Buffalo NY, where over 600 persons attended.

o ILCHV has established an Advocacy group to address disability legislative issues. They are working on developing a political candidate questionnaire for the upcoming election. They are contacting their local political officials and/or those running for political office to invite them to participate in a “Greet and Meet” seminar to be held in early October.

o ARISE organized an ADA 26th Anniversary Celebration in Syracuse. Over 400 people attended the program with speakers and then parade through downtown to celebrate the ADA.

- 28 -


o Total grassroots organizing activities engaged: 197 . Outcome exceeded. This is verified by the 2016 SSAN Yearend Report.


o Comments:

Number of oral or written public testimonies statements or letters provided, in response to a documented request, by the SSAN network (Target: 84). SSAN sites collectively provided 144 public testimonies during the past year. This exceeded last year’s activity. Examples of public testimony include:

o WDOMI submitted written comments regarding HUD’s proposed Small Area Fair Market Rent (SAFMR) proposed rule.

o STIC submitted written testimony to NYSED, in response to a request for comments on the field advisory on Least Restrictive Environment – School District Responsibilities.

o ARISE submitted written comments to Social Security Administration opposing proposed rule Implementation of the NICS Improvement Amendments Act of 2007. SSA-2016-0011 Regulations.gov. Social Security should not automatically submit names of people using a representative payee for their SS benefits to the FBI’s National Instant Criminal Background Check System.

o RCAL send a letter to Governor Cuomo in support of the visibility tax credit bill which passed the legislature this year.

o Total number of public testimonies: 144 . Outcome exceeded. This is verified by the 2016 SSAN Yearend Report.


o Comments:

Number of CILs or SCILs without a SSAN contract voluntarily involved in the participation of the network (Target: 2). Three non-SSAN centers Taconic Resources, Bronx Independent Living Services, and the Center for Disability Rights actively participated in SSAN activities throughout the year. Advocates from these centers participated in one or more NYAIL public policy subcommittees, attended at least one of NYAIL’s many training opportunities, and all three centers participated in NYAIL’s Legislative Day.

o Total number non-SSAN centers involved in network: 3 . Outcome exceeded.

This is verified by the 2016 SSAN Yearend Report.

- 29 -



o Comments:

o Formative Evaluation (what have we learned):

- 30 -


Objective # 4 – NYSCB IL FFS Consumer Satisfaction Committee

Objective 4: For eligible individuals who are legally blind, increase improved access at home or in the community and/or independence in their own lives, by supporting NYSCB to provide IL Fee-For-Services (FFS) and increase the number of CILs or SCILs in the network providing NYSCB FFS.

There were five outcomes related to this objective. Preliminary results indicate that two outcomes (or targets) were exceeded, one met, one outcome was substantially met, and one outcome was not met.


Number of legally blind consumers receiving NYSCB Fee-For-Service (FFS) (Target: 350). Based on information received from NYSCB and their central database, 305 individuals received NYSCB IL FFS during the past year. This represented almost 87% progress toward the outcome. It slightly increased from the year before (compared to 301 individuals and 86%). The service level seems to have leveled off at around the 300 mark. Total number of legally blind consumers receiving NYSCB FFS: 305. Outcome substantially met.

o This is verified by information gathered from the NYSCB Computerized Case Management System (CIS) and or Central Office/District Office records.


o Comments:

Increase in the number of CILs or SCILs providing NYSCB IL Fee-For-Service. (FFS) (Target: 2). During the past year, two CILs or SCILs were added to the existing group to increase the number providing NYSCB IL Fee-For-Services as well as other services. The Regional Center for Independent Living ((RCIL) in Rochester began providing sign language interpreter services and the Resource Center for Accessible Living (RCAL) in Kingston began providing job coaching and sign language interpreter services. Increase in the number of CILs or SCILs providing NYSCB FFS: 2. Outcome met.



o Comments:

- 31 -


Number of legally blind consumers who receive services and experience improved access at home or in the community (Target: 100). Based on information received from NYSCB and their central database/district office records, out of the individuals who received NYSCB IL FFS during the past year, 163 legally blind consumers experience improved access at home or in the community. It represents a decrease from last year’s result (183), but still well exceeds the target. When this impact is compared to the number of people served (305), it makes the statement that 53% of the consumers who received this service believe they experienced improved access at home or in the community (163/305). Total number of legally blind consumers who receive services and experience improved access at home or in the community: 163. Outcome exceeded.



o Comments:

Number of legally blind consumers who receive services and experience increased independence in their lives (Target: 125). Based on information received from NYSCB and their central database/district office records, out of the individuals who received NYSCB IL FFS during the past year, 163 legally blind consumers experience increased independence in their lives. It represents a decrease from last year’s result (183), but still well exceeds the target. When this impact is compared to the number of people served (305), it makes the statement that 53% of the consumers who received this service believe they experienced increased independence in their lives (163/305). Total number of legally blind consumers who receive services and experience increased independence in their lives: 163. Outcome exceeded.



o Comments:

Percentage of satisfied legally blind survey recipients who receive services (Target: 85%). NYSCB was working with the Center for Essential Management Services (CEMS) to conduct the Independent Living Consumer Satisfaction Survey. Unfortunately, the survey was not able to be successfully completed

- 32 -


due to a variety of factors, including the retirement of the vendor along with challenges to NYSCB staff resources. Percentage of satisfied legally blind survey recipients who receive services: N/A. Outcome not met.


o Comments:


- 33 -


Objective # 5 – CBILCO Outreach Committee

Objective 5: Improve the capacity of the IL network to address priority unserved and underserved populations from the statewide needs assessment.

It should be noted that the original objective was written for eight projects. However, unexpended funds allowed for two additional projects to be awarded for a total of ten projects. Therefore, the original targets will be stated per the SPIL, but note the adjusted number is for ten projects. These outcomes are for year two of the objective. The third year will be carried over to the first year of the next SPIL.

There were five outcomes related to this objective. Preliminary results indicate that one outcome (or target) was exceeded, two outcomes were met, and two were partially met.


1) Number of people served by identified target unserved/underserved population (Target: 480, Adjusted target: 570). Collectively the ten sites reported that they established Consumer Service Records (CSRs) with 317 new individuals over the past year. The committee noticed that this number represented 56% of the target for the year. The group felt that it could be due to due an aggressive target that may have been set too high. It was felt that Brad should mention this fact in an October email communication to the CBILCO projects and encourage them to improve CSR productivity during their final year. Also, the issue of monitoring came up as a concern that would have to be addressed in the new SPIL. The SILC would have to increase its involvement for SPIL projects and develop protocols for how to communicate with them during the monitoring process. The Outreach Subcommittee identified four projects for Brad to reach out to. He touched base with them which added clarity to information not found in the reports. A fourth project ended up ceasing their project due to difficulties maintaining a project coordinator.

o Total number of people served by identified target unserved/underserved population: Outcome partially met.

This is verified by July-September 2016 quarterly reports submitted by CBILCO vendors, the CBILCO Excel spreadsheet and minutes.


- 34 -


o Comments:

Number of community organizational contacts related to targeted populations that are developed as a result of outreach efforts (Target: 40, Adjusted target: 50). Collectively the ten sites reported that they made 67 community organization contacts as a result of outreach efforts related to their target populations. The projects were active in the first quarter, minimal in the second, and then picked up in the third quarter. Examples of selected community contacts from the projects include: Elmira and Horseheads School Districts, Syracuse University Parent Assistance Center, Cortland Works Career Center, Cortland and Homer High Schools, Acacia Network, Inc., Montefiore Community Center, Niagara Orleans BOCES, Clinton County Office for the Aging, Wiggletown Taxi, Clinton County Department of Health, Rockland Youth Bureau, Southern Tier Veterans Support Group, and Wounded Warriors Project.

o Total number of community organizational contacts related to targeted populations that are developed as a result of outreach efforts: 67. Outcome exceeded. This number exceeds both the SPIL and adjusted targets.



o Comments:

Amount of new funding secured toward self-sustaining programs (Target: $66,666, Adjusted target: $83,332). Collectively, six sites have worked on developing self-sustaining opportunities, while two sites successfully secured funds during the year. Two projects attracted $43,297. WNYIL attracted $200,000 from the Tower Foundation for its youth with disabilities project, with a yearly disbursement schedule. They have access to approximately $41,047 annually. ATI Cortland obtained $2,250 in grants to support their youth with disabilities project from two different sources. The committee noted that while this target looks at funds attracted to the project, an assessment of sustainment won’t be made until a year after the projects end – as noted in the next target.

o Total amount of new funding secured toward self-sustaining programs: $43,297. This represents 52% of the adjusted target. Outcome partially met.



- 35 -


o Comments:

Number of self-sustaining programs (Target: 0) The measure for this target is zero and no project has yet to attain a “self-sustaining” status after the completion of the second year of activity. Remember that while this is the third year of the SPIL, these projects are in their second year of implementation. Total number of self-sustaining programs: 0. Outcome met.

o This is verified by July-September 2016 quarterly reports submitted by CBILCO vendors, the CBILCO Excel spreadsheet and minutes.


o Comments:

Number of “how to” technical assistance manuals submitted to NYSILC (Target: 0). Just about every project noted progress in their quarterly reports on their “how to” technical assistance manuals. The group felt that NYSILC should convene a group of the committee the content of the “how to” manuals for the CBILCO projects. This should include how to evaluate the manuals and best distribute and post them for the benefit of the network. The council must send out an October communication to the projects that provides guidance related to the manuals. Total number “how to” technical assistance manuals submitted to NYSILC: 0. Outcome met.

o This is verified by July-September 2016 quarterly reports submitted by CBILCO vendors, the CBILCO Excel spreadsheet and minutes.


o Comments:


Objective # 9 – Database SPIL Committee

Objective 9: Designate funds for the purposes of developing technical specifications and establishing a database to compile, analyze and interpret data from the statewide network.

There were four outcomes related to this objective. One was met, one partially met, and two were not met.

- 36 -


In hindsight, a more realistic timeframe should have been foreseen related to the development of database product.


1) Percentage of completion of a multi-purpose, interface database design (Target: 100%). The Database Work group met four times during the year. At the 3/31/16 meeting, the group met to review three vendor proposals for the database and interface based on the project description. In the end, the group felt that the proposals required additional information. However, the group was only interested in interviewing the state university and Albany based vendors. Brad thanked the third vendor for their time and arranged dates for follow up discussions with the two vendors in early May. The group is generating questions to ask each vendor in advance of the calls. During the 5/2/16 meeting, the members met with a local vendor to follow up on questions related to their proposal. They were pleased with the details of the proposal including the scope of work. The cost was within the $75,000 budget. Ongoing maintenance was $15,000 a year and including hosting costs and revisions to the database. Brad will follow up with credentials and work examples. At the 5/3/16 meeting, the group met with a statewide vendor to address questions related to their proposal. Members valued that the vendor had advanced knowledge of social research. However, they identified instances in the proposal that addresses only minimal requirements. In order to achieve what members felt was four times during the year, appropriate in terms of an interface, mobile devices, and a mapping option will require a significant amount of additional funds. Ongoing maintenance would cost $15,000 a year. It did not include hosting (extra) or changes to the system. The vendor indicated they will follow up with additional materials. During the 8/4/16 meeting, the group met Center for Health and Social Research (CHSR) staff who provided a live demonstration of a potential online product. The following link provides audio to the webinar session: http://www4.buffalostate.edu/centers/chsr/media/2016-08-04%2010.12%20NYSILC%20Webinar.mp4. The workgroup ultimately felt the proposal submitted by the local/regional vendor Jeremy McGowan Enterprises Corp had the best potential and discussed it at a meeting with ACCES-VR on September 20, 2016. Next steps were then discussed related to the project. The conceptual design of the project was complete. It would need to be addressed as a network along with confirmation of data fields and current versions of center databases.

o Percentage completion of data interface design: (100%) . Outcome met. This is verified by the Database Workgroup meeting minutes and

materials.

o Value added:

- 37 -



o Comments:

2) Percentage of CILs/SCILs contributing to the multipurpose, interface database design (Target: 60%). Four members of the work group are CILs/SCILs and participate actively in discussions and online feedback that contribute to the database’s design and development (STIC, WNYIL, ATI Cortland, BILS, and CDCI). It will be crucial for the project that and education piece be developed and centers understand the nature of the project as confirmation of the data fields and center databases occur going forward.

o Percentage of CILs/SCILs contributing to design : 16%. Outcome partially met.

This is verified by the Database Workgroup meeting minutes and materials.

o Value added:


o Comments:

3) Percentage of 21 st Century Data and Collection Model Workgroup CILs/SCILs that will engage in testing the new database design and framework (Target: 75%). No work has taken place on this step of the process. It is premature. The design of the product and identification of a vendor was recently achieved.

o Percentage of 21 st Century Data and Collection Model Workgroup CILs/SCILs that will engage in testing the new database design and framework: 0%. Outcome not met.


o Value added:


o Comments:

4) Percentage report completed by NYSILC and the DSU (ACCES-VR) based on statewide 2012 CIL/SCIL data (Target: 100%). No work has taken place on this step of the process. It is premature. The design of the product and

- 38 -


identification of a vendor was recently achieved.

o Percentage report completed by NYSILC and the DSU (ACCES-VR) based on statewide 2012 CIL/SCIL data: 50%. Outcome not met.


o Value added:


o Comments:

o Recommendations for next year:

- 39 -

Documents

Background - NYSILC · Web viewObjective # 1 – NYSILC Operations16 Objective # 2 – SSAN22 Objective # 4 – NYSCB IL FFS29 Objective # 5 – CBILCO31 Objective # 7 – CIL Coaching33