TAI Practical Strategies to Support Data Collection and ... · data collection; •Participants will ... Notes from Slide 11: Data: Good data isn’t just a bunch of raw numbers

PRACTICAL STRATEGIES TO SUPPORT DATA COLLECTION

AND ANALYSIS

1

WE ARE GLAD YOU ARE HERE!

ITACC: 2017 AIDD Technical Assistance Institute2

LEARNING OBJECTIVES

• Participants will recognize basic concepts related todata collection;

• Participants will understand data needed to report onnew performance measures;

• Participants will explore data collection activities andstrategies.


4 THINGS TO KNOW

• Know why you want to evaluate a project/activity.

• Process of evaluating progress towards goals

• Collect and Use Data

• Accountability for results

• Clarity of outcomes


Notes for Slide 4:

Before collecting data – Know why you want to evaluate a project/activity.

Logic model identifies desired outcomes, resources, and activities necessary to accomplish the outcomes.

• Process of evaluating progress towards goals:

o Align structures, processes, and routines to enable a methodical and routine evaluation process between work and desired outcomes.

• Collect and Use Data

o Establish and implement routines and processes for collecting, analyzing, and monitoring data

• Accountability for results

o Making decisions to continue, improve, or end practices based on data; communicating results with internal and external stakeholders

• Clarity of outcomes

o Communicate priorities, set ambitious, clear, and measurable goals and outcomes with aligned strategies and activities.

5

CONNECTIONS WITH STRUCTURES AND PROCESSES

Systematic approach to ensure

quality and progress towards

goals

Aligning structures, processes,

and routines

Logic Model

Evaluation Plan

annual work plan – data

collection


COLLECT AND USE DATA

The collection of data can inform continuous improvement, provide feedback, and help guide decisions.


Notes for Slide 6:

Collect and Use DataEstablish and implement routines and processes for collecting, analyzing, and monitoring data.

There is a resource on the itacchelp.org website - http://itacchelp.org/wp-content/uploads/2015/12/2015-TAI-Evaluation-FAQ-3.pdf

8

http://itacchelp.org/wp-content/uploads/2015/12/2015-TAI-Evaluation-FAQ-3.pdf

ACCOUNTABILITY FOR RESULTS

• Decisions based on results

• Improve

• Adjust

• Terminate

• Incentives

• Additional or continuation funding

• Communicating results

• Internal and external stakeholders


Notes for Slide 7:

Making decisions to continue, improve or end practices based on data; implementing incentives tied to performance; and engaging and communicating results with internal and external stakeholders.

10

CLARITY OF OUTCOMES

• Establish Priorities

• Communicate priorities

• Set ambitious, clear, measurable goals and outcomeswith aligned strategies and activities.


Notes for Slide 8:To ensure the Council is getting what they want from projects and activities, it is important to establish priorities for the activity and to communicate the expectations. Of course, setting clear, measurable goals and outcomes that align well with the strategies and activities will help

12

HAVE YOU EXPERIENCED THIS WITH GRANTEES?


Notes for Slide 9:

When councils engage other entities to help accomplish goals and objectives – and changes are made to the initial plans, don’t forget to adjust the evaluation component as well – ITACC staff frequently hears about data being provided to Councils that does not speak to the overall outcomes of the funded project/activity – having a process in place that helps grantees understand the data that is needed to measure impact is critical!

14

MEASURING IMPACTS AND RESULTS


Notes for Slide 10:

Let’s take a look at the new performance measures – the new measures call for connection between investments, outputs, and the outcomes of the activity – this translates into results and short-term impact, intermediate impact, and ultimate impacts of Council activities

16

PM INFODATA = IT’S NOT JUST NUMBERS

• Data = facts and statistics collected together for referenceor analysis.

• Do you know what you need to know?

• Think Impact, Impact, Impact!!

• Lessons learned:

• Training for sub grantees is very important

• Councils need to take the lead

• Request more than you think you’ll need

• Continuous monitoring and follow up for clarification

• Self reporting and incomplete survey’s

• Use examples and tools from states who’ve been doing it


Notes from Slide 11:Data: Good data isn’t just a bunch of raw numbers – it’s a compilation of facts based on a clear understanding of WHAT you are tying to find out. The results should tell a story. Do you know what you need to know?These PM’s aren’t new. Although they keep changing slightly, the basic concept of what you need to collect isn’t. Make sure you really read and think through these PM’s so you know what you need to collect.Lessons Learned: Initial training is vital. Talk to the grantees about the new way we need to collect and report on data.Put the work in up front - look ahead at what you need to report in the PPR, then track back to figure how you will get that information from the grantees in a way that’s clear and concise. For us it was creating our own reporting templates which will get to in a min.Take the lead in telling sub-grantee’s what you need to know and how you want it reported. I tell the grantees if they question whether info is relevant – put it in and I will determine that because I’d rather have more than not enough.Continuous monitoring = allows you to make sure that you’re getting the data, and that it’s relevant to what you need to report in the PPR. Follow up with sub-grantees is very important.Make sure you get clarification.Looking at surveys = not everyone completes every question on a survey! You have to calculate percentages based on the number of people who answered a question – not based on the numberof people who completed the survey!

18

CAN'T HAVE ONE WITHOUT THE OTHER

•Logic Model

•Work plan

State Plan

•Excel spreadsheets

Data Collection

•DD Suite

•Surveys

•Sub-grantee reports

•Impact Statements

•Personal Stories

•Sign in sheets

•Focus Groups

•Conversations with sub-grantees

•Narrative

•Performance Measures

PPR/IMPACT

How We Show Our Worth


THE OREO EFFECT

STATE PLAN DATA PPR OUR IMPACT


Notes from Slide 13:

The Oreo effect. The filling is the data collection, it’s what holds the top and bottom wafers together. Without it, you’d just have chocolate wafers, which are ok, but it’s just not an Oreo.

21

LET’S LOOK AT SOME TOOLS



With change – Council’s have the opportunity to revisit the evaluation tools that will best meet the

Councils needs…

23

SOUTH CAROLINA

• Quarterly Progress Reports

• Surveying Process

• Monitoring Visits

• Grantees can enhance minimum reporting

requirements to best-fit project/activity

• Ultimate question from Council staff is: “how are you

measuring the impact of your programming?”


Notes for Slide 15:Adapted AIDD performance measures into quarterly progress reporting and surveying processes with follow-up provided by monitoring visits.Program monitoring site visits complement the sub-grantee reporting process in addition to ensuring compliance and integrity to Terms and Conditions, relevant state and federal rules, and original grant Request for Proposal.While SC DD Council provides the minimum requirements for survey data collected, we encourage sub-grantees to modify our tools to best meet their program needs (surveys and other impact-gathering tools may be modified only—quarterly progress report is standard).Most common question from DD Council program staff to sub-grantees is “how are you measuring the impact of your programming?”

• Sub-grantees opting to modify our survey tools, must consult with programming staff to ensuredata being collected is relevant to all involved.

• Program monitoring is best when program staff can see project activities in action. Get to meetpeople being impacted by activities and raise awareness of SC DD Council (we are often confusedwith our state’s DD service provider, DDSN)

• Most systems change projects include a committee or workgroup that DD Council is usually invitedto be a part of (not a requirement, though)

• Impact has a variety of uses and meanings in SC. Can mean short term outcomes (e.g., participantsdeveloped a peer network of support while learning about new employment benefits) but also long-term (e.g., participants organized themselves into a local “advocacy” group in Charleston and speakregularly at City Council meetings. This contact resulted in a member being asked to join theMayor’s Advisory Committee a year after the individual participated in the original grant activities).Projects working directly with individuals and families are asked to assess how their grant activitiesare affecting participants lives outside of grant activities. EG, a new skill or knowledge area might beintroduced to participants, but how are the participants using these new skills and how have theirlives in general been affected by being a part of sub-grantees’ activities?

25

Tools We Use in South CarolinaQuarterly Progress Report (QPR) Handout

• Quarterly Progress Report (QPR) (Handout)


http://www.scddc.state.sc.us/Reportandreimb.html


Word document created with “developer tools tab,” “design mode,” “controls panel.” including “click text” boxes to help guide sub-grantees through the QPR. The document is “locked” so sub-grantees cannot accidentally modify it and can only enter info in the “click text” boxes. There are instances in which a sub-grantee needs to add rows to the objectives section of the QPR, in which case, the sub-grantee must come to program staff. This is good because some sub-grantees include a lot of objectives to their programming and this affords an opportunity to discuss realistic expectations for a 12-month program—more than six objectives to a grant project is a “rose flag” (not a red flag) to program staff to take a closer look at the sub-grantees programming expectations so they don’t freak out (and Council program staff doesn’t freak out, too).

27

QPR – THREE PRIMARY FOCUS AREAS

• Number of people served

• Numbers – quantitative

• Grant Activities

• Narrative – qualitative

• Partners and System Change

• Numbers and narrative



1. Number of people served (numbers, quantitative)• Individuals with intellectual and developmental disabilities

o Sub-grantees are trained in the DD Act definition of Intellectual and developmental disability• Family members• Other: sub-grantee is asked to describe who participated in activities (direct service professionals, teachers, law enforcement staff, doctors,

lawyers, etc.)• 1: People served: Only those participating in organized grant activities (conference exhibiting, publicity, awareness raising, and similar events

should never be counted in number of project participants). Sub grantees are asked to show how they are tracking participants and gatheringinformation on disability.

2. Grant Activities (narrative, qualitative)• Evidence base: best and/or promising practices• Activities narrative

o Sub-grantee reflects on activities only, specifically, what they expected to happen versus what actually happened.o Helps SC DD Council staff discern whether or not the sub-grantees’ activities are contributing to improvements in existing or creation of

new best and/or promising practices (AIDD PM).• Quarterly narrative

o Collect success stories, challenges experienced, lessons learned, collaborations, and unexpected outcomes.o This section helps drive our focus for DD Council newsletters and for annual Program Performance Review.o Particularly important to capture nuances in data not reflected in numberso Also important in helping sub-grantees to avoid duplication in reportingo 2: Grant activities: focus on the activities, not the impact. In addition to program staff evaluating whether or not the sub-grantees

activities are leading to improvements in best/promising practices, this section helps program staff evaluate whether or not the sub-grantees’ activities have clear links to the overall program outcomes. Quarterly narrative: focus on impact. Sub grantees areencouraged to share success stories and lessons learned. Program staff gathers questions for monitoring visits (from the whole QPR,but particularly this these narrative sections). Duplication—some grantees partner on multiple projects including conferences. Programstaff has to ensure participants are being counted only once.

3. Partners and Systems Change• Draw sub-grantees’ attention to changes in policy and procedure both internal and among project partners and stakeholders• Some sub-grantees’ projects are working to directly influence law and regulations• Numbers are requested as part of AIDD PM, and SC DD Council also asks for descriptions of policy and systems change work• This section is important in helping program staff work

Sub-grantees often influence policy and procedure and do not realize it especially processes internal to their own organization.--Descriptions are requested to follow up on during monitoring visits.-- The prepared survey tools provided by DD Council are not set-up to gather systems change data, they are geared towards projects working directly with people. Systems change work is more nuanced and, therefore, we work with individual systems change projects to create assessments for impact.--Review contexts, use QPR instructions.

29

TRAINING FOR GRANTEES

Sub-grantees receive yearly training on QPR and

surveying.

Important Contexts for Sub-Grantee QPR Training

1. Best practice verses promising practice

2. Organized grant activities

3. Advocacy in our SC DD Council (and DD Network)

context

4. DD Act IPSII Outcomes

Questions before we

review survey tools?


SC SURVEY TOOLS

• SC DD Council prepared two survey tools to be used by sub-

grantees.

• Sub-grantees are trained that these include the minimum

requirements for surveying and they are encouraged to

modify surveys based on their program needs and focus.

• Sub-grantee projects not working directly with individuals

and families or who are working on systems change projects

must work with DD Council program staff to create a

combined satisfaction and outcomes survey specific to their

program.

http://www.scddc.state.sc.us/Reportandreimb.html


Notes for Slide 19:--SC DD Council’s State Plan Evaluation Plan states we will measure four key areas of satisfaction: behavioral, affective, emotional, and cognitive satisfaction. Therefore, program staff TA ensures that sub-grantees creating their own satisfaction survey is relevant to these measures.--The outcomes survey collects basic demographic data and measures the level of advocacy (keeping in mind a broad definition of advocacy/self advocacy) increases as a result of the project's activities. Outcomes survey mirrors AIDD PMs.--These templates are word documents that are easily edited. Many sub-grantees modify them into Survey Monkey or other online platforms.

32

SC SURVEY (HANDOUT)


ITACC: 2017 AIDD Technical Assistance Institute

SC SATISFACTION SURVEY

• SC DD Council’s

satisfaction

survey collects

quantitative and

qualitative data

• Key Focus on

behavioral,

affective,

emotional, and

cognitive

satisfaction

Affective (like/dislike) and behavioral satisfaction:May measure how something

specific affected one’s behavior

Behavioral satisfaction:May measure intent or similar

behavior

Affective (like/dislike) and Emotional satisfaction:Measures satisfaction overall or on

specific topics

Cognitive Satisfaction:Measures judgment

34

http://www.scddc.state.sc.us/documents/DDC Satisfaction Survey.docx

Notes for Slide 21:There are overlaps in affective/behavioral; affective/cognitive satisfaction because affective (liking/disliking something specific) measures indicate how a specific item affects one’s behavior (behavioral) or judgment (cognitive)Affective (like/dislike) and behavioral satisfaction:May measure how something specific affected one’s behavior

• Would you recommend this event to a friend?Behavioral satisfaction:May measure intent or similar behavior

• Would you participate in another event provided by this organization?Affective (like/dislike) and Emotional satisfaction:Measures satisfaction overall or on specific topics

• How satisfied are you with the way the material was presented?• How satisfied are you with the subject matter covered?

Cognitive Satisfaction:Measures judgment

• What was your favorite part of the event?• What was the most useful thing you learned?• How would you change this event?

35

Outcomes Survey

• Adapts AIDD PMs not included in the QPR to collectdata directly from individuals and families.

• Sub-grantees working on systems change projects mustwork with DD Council program staff to create atargeted data collection tool.

Data Measures that must be included (AIDD PM’s)

Advocacy, self-advocacy, and self-determined behavior

Individual or family member’s ability to speak up forthemselves

Board, committee, coalition involvement

http://www.scddc.state.sc.us/documents/DDC%20Outcomes%20Survey.docx


http://www.scddc.state.sc.us/documents/DDC Outcomes Survey.docx




Notes for Slide 22:• These outcomes surveys are intended for individuals with intellectual and developmental disabilities

and their family members.• Some sub-grantees feel this survey is not appropriate for people with severe disabilities in which case

we partner with them to work on data collection tools appropriate for their audience.o Strategies include one-on-one interviews or group interviews. Have included sub-grantee

project staff and DD Council program staff or, my favorite, peer lead group interviews.--Advocacy: Sub-grantees are trained that DD Council takes a broad view on advocacy, selfadvocacy, and self determined behavior. Definition provided on the survey and sub-granteesare encouraged to emphasize the “speaking up for someone else or yourself” aspect ofadvocacy. Attempt to measure a before and after, but we do not encourage pre and post-testing because you don’t know what you don’t know.

• Advocacy, self advocacy, and self determined behavioro Both learning how to speak up and increasing speaking up whether in a formal way or in a

general wayo Advocacy, self advocacy, self determined behavior including IEP meetings, providing legislative

testimony, Advocacy Day for Access and Independence, Community Engagement Day• An individual or family member’s ability to describe what they need or want either in general or in

formal services (depends on the project)• Board participation

o Maybe an individual or family member gained membership. If not, we ask what kind of boardthey would like to be a part of and make connections when appropriate

37

Systems Change Outcomesand Impacts Data Collection

• Sub-grantees and Council staff work together to create a surveythat measures:• Satisfaction

• Project specific goals

• Existing policies

• Other measures

• Surveys are anonymous and administered online (in general)

• Purpose:• Gather information on how systems operate currently

• What level of appetite exists to move systems in a defineddirection

• What policies and laws can be targeted for change

• Future areas of focus for the DD Council (in general)

• Data collected may be used to amend or influence updates toState plan and annual work plans


Notes for Slide 23:

Use SDM survey as an example.--What steps should stakeholders take to further SDM in SC?--What specific activities can your organization do to further SDM in SC?--What policies exist in your organization relating to guardianship and SDM?

39

NEVADA

• Collection

• Make sub-grantees do the work for you

• Individualized reporting templates and surveys

• Provide in depth training when grant is awarded

• Sub-grantee checklist

• Calculation

• PM Spreadsheet

• Impact

• Council PM’s from 5 year plan

• Impact of Activity

(sub grantee report template)

• Satisfaction Survey’s

• Personal stories


Notes for Slide 24:COLLECTION

• We represent the smaller states who may not have 10+ sub-grantees. We also do a lot of ourprojects in-house.

• Sub grantee reporting templates and surveys: I create them based on their application – howthey said they would meet the specific requirements of the Goal/Objective in our 5 year stateplan (Review sections of template) add what I need to know for the PPR.

• PM’s - we found that we get information we can use that we might not have been able tocollect if we limited the PM’s we had them report on).

• Review sections of surveys (Everything we do is based on our 5 year plan objectives. We getour I/DD and family satisfaction data from the surveys. Each grantee is required to completesurveys for each activity and we also do surveys for Council or “in-house” activities. Eachsurvey has the correlating objective printed on it. That info ends up in our Sat Fact. Surveyresults tab on our spreadsheet.)

• It’s more work for me up front but ends up less on the back end.CALCULATION

• Review PM spreadsheet (how do I calculate my numbers?)IMPACT

• We’ve come full circle back to the 5 year state plan! (cartoon – if you’re doing it right, it shouldfeel like your 5 yr state plan is following you around! And although we may all take differentdirections, we all end up back at the same place.

• Review Council PM’s in PPR41

TEXAS

• Average 30-35 grants each year

• Multiple start and end datesdepending on project- impactsPPR reporting year

• Rely on grantee to identifyoutcomes of individual projectsand annual workplan

Current Data Collection:

• Numbers related to specificperformance measures

• Additional data from projectnarrative in DD Suite


TEXAS CONTINUED

Shifts in Data Collection:

• “Assigned” performance measures based on stateplan goal (can add others as project allows)

• Sample tools for satisfaction and advocacyactivities

• One-on-one TA for data collection methodology

• Collection of personal stories

• Continue to rely on project narratives


QUESTIONS


YOUR TURN! – NETWORKING ACTIVITY

• Let’s use our remaining minutes to share datacollection ideas and strategies with each other

• What did you hear about today that you may want touse or modify for use?

• What are you doing that others may find helpful?


Documents

TAI Practical Strategies to Support Data Collection and ... · data collection; •Participants will ... Notes from Slide 11: Data: Good data isn’t just a bunch of raw numbers