Using the ODSS to Report Progress and Impact

Iowa SPF-SIG InitiativeMay 30, 2013

• Part I: Reporting Your Results• Part 2: Using ODSS Data to Answer

Evaluation Questions• Part 3: Using the ODSS to Create

Graphs

Part IReporting Your Results

What is providing a report?

• Presenting data from the evaluation to those involved in the initiative

• Usually presented in graph form• Should be done on an ongoing basis; can

also be done at the end of an evaluation

Why provide a report?

• To assess progress towards goals• To see where more energy may be needed and

where too much energy is being spent • To focus on and celebrate cumulative

accomplishments • To show funders your accomplishments and

provide them the opportunity to help re-direct the initiative

When should you report out?

• Your staff/volunteers should get feedback at regular intervals, especially early in the initiative's development.

• You might want to have feedback sessions on a quarterly (four times a year) basis.

Present the information

– Introduce the information (definition, example, and why it's important)

– Give a detailed example from recent accomplishments – Point out what's positive – Make note of any trends– Discuss re-direction of efforts – Review examples on graph– Note evidence of progress– Ask for comments or questions from the group– If the measure doesn't show recent growth: Show any

previous growth

Overall tips

• Present data as a good thing• Focus on the positive• Convey the following:

– optimism about prospects for success (if appropriate)– the need for changes or adjustment (if appropriate)– a shared vision of the initiative as an effective catalyst

for change

Reasons for informing the public at the local level:

• Raise awareness • Attract resources• Promote registration • Lobby for ordinances or program changes• Provide accountability

Three tips for making sure your findings aren't ignored:

• Give your information to the right people!• Address issues which those people think are

important• Be sure the information is presented in time

to be useful and in a way that's clearly understood

What are some key audiences for the data?

• Staff• Volunteers• Supporters in the community• Funders• Your target population• The general public

Part 2 Using ODSS data to answer

evaluation Questions

Challenges

• A key challenge in evaluating comprehensive community health initiatives is how to document and illustrate the unfolding of the intervention; that is, the multiple changes in the environment that can lead to widespread behavior change and improvement in population-level outcomes.

Illustrative evaluation questions• What actions have we taken to bring about new/modified

programs/policies/practices?• Is the initiative a catalyst for change?• How are our efforts contributing to desired outcomes?• Which intervening variables are most important in our efforts?• How are we bringing about changes among community sectors?• Are we seeing changes in underage drinking?• What policies were changed or new policies enacted to restrict alcohol

use?

Linking graphs to evaluation questions

• What new/modified programs and practice changes occurred in the community?

How are we contributing to changes in our outcomes?

Illustrative examples of programs and practices

• In collaboration with SPF SIG and the police department, the County passed laws regulating happy hours and sales of alcohol

• For the period of August 7-August 14, 2012, the County SPF SIG ran the underage drinking radio ad for the first time on KUOO 5 times/day for 7 days consecutively.

How are we bringing about changes among community sectors?

How are our efforts contributing to desired outcomes?

On which intervening variables are we focusing our efforts?

On which strategies are we focusing our efforts?

Part 3Using the ODSS to Create Graphs

Creating Graphs

• Using online graphs, community and evaluation partners can examine the rate of community/system changes over time, or allow for systematic reflection on the proportion of changes occurring in different sectors.

Creating Graphs• To begin, click on the CREATE Graph button on your

main ODSS page.

Creating New Graphs

• The next page will display:

Creating New Graphs

• Create a new graph:– Used when creating a graph for the first time or when creating a

graph with a unique set of data.• Graph from template:

– Used when creating a graph that will use the same settings as a previous graph. Once a graph is created, the option exists to save it as a template to be used for future graphing. (For example, once you have created a graph of Accomplishments, you can save it as a template, and use this template for any future graphs of Accomplishments.)

Creating New Graphs

• Use the drop-down menu to identify the data you want to graph.

• Note: This drop-down menu will feature any ODSS data or list data available on the site.

Example of Community Change Cumulative Line Graph

• When creating a CC cumulative line graph, you would select the following options from the drop-downs and then Create Graph:

Example of Community Change Cumulative Line Graph

• You will then be presented with a graph for which you can change the date range, date display, labels display, and graph type

• Date Range:– Adjust the range of data displayed by changing the dates shown. You can

type the dates in, or use the calendar that appears when you click within the box.

• Date Display:– For a broader or narrower view of the data points throughout the designated

date range, you can change the Date Display to reflect data points by Day, Month, Quarter or Year.

• Labels Display:– Display or hide data labels on the graph to help identify the values shown in

each data point of the data series. If using a pie graph, the option will exist to Show Percentage.

Types of Graphs• If you would like to view varying presentations of the data, you can

modify the graph type by selecting from the Graph Type icons.• Line Graph:

– Line graphs are used to track changes over periods of time or cumulative changes.

• Pie Chart:– Pie charts are best to use when you are trying to compare parts of a

whole or show a distribution of categories. • Bar Graph:

– Bar graphs are used to compare data between different groups or to track changes over time.

– When trying to measure change over time, bar graphs are best when the changes are larger.

Edit Graph• When you click the Edit Graph button, this will open a new window with

options to Edit Basic Information, Edit Data, Add Additional Data, Add Additional Sites, and Add Comparison Graph. Simply rollover the category you want to edit, and then type in the new information.

Edit Basic Information• Title

– Update the title of the graph. – Note: The default title will always be the name of the data selected for the

graph (e.g., Accomplishments, Participant Data).

• Date Range– Adjust the range of data displayed by changing the dates shown.

• Date Display– To review a broader or narrower view of the data points throughout

the designated date range, you can change the Date Display to reflect data points by Day, Month, Quarter or Year.

• X and Y Axis Label– Update the label for the X Axis and Y Axis.

• Labels Display– Click here to either display the data labels on the graph, or keep them hidden.

• Graph Type:– If you would like to view varying presentations of the

data, you can modify the graph type by selecting from the Graph Type icons.

• Edit Data

• Legend– This field allows you to modify the legend of the graph. – Note: The default for the legend will automatically be the name of

the data selected for the graph (e.g., Accomplishments, Indicators, etc.)

• Data Source– This is the source of your data (e.g., Accomplishments, Indicators,

etc.)

• Observer– Observer 1: Primary Data– Observer 2: Reliability Scorer Data– Observer 3: Calibration Scorer Data

• What specific information do you want to graph?– This field is where your data is coming from, i.e., what you are graphing (Code, in this

example). You can also filter by Code when selecting the specific information to graph

• Add Additional Data– Select from the data sources on your site to include additional data in your graph.

• Add Additional Sites– Select additional sites from within the site collection of your WorkStation to layer data

in your graph.– Please note that you are able to select multiple sites (e.g., different programs or

locations) and series (e.g., different data sets) from which to graph.

• You also have the option to add a comparison graph. For instance, if you want to compare Accomplishments from this year to the previous year, you can choose last year’s graph from the Select Comparison Graph drop-down menu and hit Save. This will return you to the graph display, and you will see the graph that you just created on the left side, and the comparison graph that you just added on the right.

• Once you have added a comparison graph, you can also edit this graph, by selecting Edit Comparison Graph. If you decide you no longer want the comparison graph, you can simply click the Edit Graph button again, and under Add Comparison Graph, you can choose No graph under the Select Comparison Graph drop down menu.

• Once a comparison graph is selected, you will be able to view the two graphs stacked within the window that appears.

• During the graph creation process, you may find that you periodically create graphs with the same set of criteria. To help save time during this process, you can create a template from a graph. You can select Manage Templates from either the web part menu or Graph Management:

• The Create Template button will allow you to create your new template. This page will also show all existing templates that you can choose from to create a new graph.

• Once you have created a template, the next time you create a graph, when prompted Create a new graph, or Create graph from template, you can select the template created from the drop-down menu.

• To view all the graphs that have been created on the site, click the VIEW Graphs button on your main ODSS page.

Viewing Graphs• This window will display all available graphs on the site.

Viewing Graphs• You can view additional details about each graph by placing your cursor

over the specific graph. You will be able to view the Date Range used when creating the graph, the date the graph was created, and who created the graph. If the graph has been modified, it will display the details from the most recent modification, rather than the initial creation.

Adding Key Events to Graphs• Key events are occurrences in the external

environment in which the initiative is not involved and of which the group has no control.

• These events may include a change in a mayoral administration, an alcohol-related death of a teenager, or the passing of an alcohol-related ordinance not brought by the initiative.

• To document a key event, locate the Key Events web part on your ODSS and select “Create Key Event”

• On the next page, put the date that the event occurred and a label of 35 characters or less (including spaces). It is also helpful to include a description of the event (unlimited number of characters), although this field is not required.

• After you have filled out the fields, you may click Save & Return if you would like to return to your ODSS homepage, or Save & Add Another if you would like to continue entering key events. You also have the option to click delete or cancel

Graphing Key Events• After you have created a graph, using the steps previously

shown in this presentation, select Edit Graph

• Next, scroll over to the Add Additional Data tab and under the Data Source drop-down, select Key Events

• The key events that you previously entered will appear and you can select any or all of the events that you would like to include on your graph

• After selecting your key event(s), click Add Data

• Your new graph will be populated to display each key event as a circle and each time you scroll over a key event, its label will be highlighted in black at the bottom of the screen

Supports for Improvement• The Supports for Improvement link provides helpful

guidance and links for solving problems and dilemmas documenters may be encountering in their work.

Supports for Improvement• Here, you will be directed to resources provided by the

Community Tool Box to support your effort.

Reflection Questions• Reflection questions are tailored questions that encourage

documenters to think carefully about, better understand, and communicate the meaning of their data.

Reflection Questions• The Reflection Questions link offers questions such as

“What are we seeing? What does it mean?