Military Family Services ProgramParticipant Survey

Training Presentation

MFSP Accountability Framework The System

The method (survey) used to collect data to measure the performance indicators

How Was the MFSP Accountability System Developed?

Involvement of field throughout the process

With intention to create value at centre level and at MFSP level

With an emphasis on how to improve

Why a Survey Approach?

Our main choices are to: – observe– ask

Surveys are an efficient method of obtaining information about experiences, opinions, past behaviors– these cannot be observed

Surveys are easy to analyze

Evaluation Defined

The systematic collection of data to answer questions about a service or program

Evaluation – A Snapshot in Time

The information provided by an evaluation is not the program

The purpose is to generate a fair, accurate, representative picture of those aspects of the program that are of most interest

Evaluation Provides Information For

AccountabilityProgram Development

Marketing

Key Evaluation Questions

Who are your participants?What services, resources are available or

provided?What is the quality of these services?What benefits are obtained by your

participants?

How will Information be Used?

Responses within each centre will be aggregated

Centres can look for trends in responses and use information to better understand what works well and what needs attention

DMFS will also aggregate data on national level and use information to manage MFSP

Impact on Funding

The survey will not have a direct impact on funding

It is one of many sources of information that are taken into account in the management and funding of centres and of the MFSP as a whole

What are the Elements of the MFSP Accountability Framework?Financial accountabilityMOUNeeds assessmentAuditsFormal site visitsEvaluation of services

(see Parameters for Practice, pp. 15-16)

Accountability Framework at a Glance

Service

Category

Service Components

Objectives Indicators

Personal development and community integration

5 22 34

Child and youth development and parenting support

5 19 26

Family separation and reunion

1 7 11

Prevention support and intervention

4 14 22

TOTAL 15 62 93

The Accountability System Aim was to create feasible system that

provided meaningful information for a reasonable effort

A selection from the accountability framework– Based on field based workgroup priorities– DMFS management priorities

17 objectives and indicators selected– All service components represented (two have 2

indicators each)– Some relate to process; some to outcomes

The Accountability System (2) In addition 4 items were selected from

principles of family support as described in MFSP: Parameters for Practice

11 structured items look at participant characteristics and participation patterns

3 open ended items to allow respondents to expand on any answers or provide additional information.

Overview of the Questions in the Survey

21 structured rating items11 structured questions describing

participants and participation3 open-ended questions

How does the Survey work?

Items are written to obtain information about the experience of participants, – not a direct reflection of centre

services (services may be provided directly, indirectly or by others)

Respondents should think of current posting when answering questions

Why Collect this Information?

Implementation of the service accountability framework in MFSP: Parameters for Practice (pp. 33-52)

It is hoped that this system will replace BSI and NID

Will provide more meaningful information

Evaluation Planning DMFS establishes

evaluation requirements common to all MFRCs

Centres determine their own additional evaluation needs

Centre Evaluation Committee or equivalent

Develops a centre evaluation plan that integrates local and corporate

evaluation requirements

Example of a Local Evaluation Plan

Required MFSP survey

Interested in more detail about Assessment and Referral component—develops own survey

Conducts focus groups to get more

detail and context for survey results

Can use items in accountability

framework and/or create own

Possible Uses of Evaluation Information

Information

Funding application

Service delivery planning

Organizationalplanning

Activity Listing

Evaluation Plan

OtherInformation

Improvements

in Services

Improvements In Programs

New Programs

Strategic Planning

Staff Training

Facility Issues

Advocacy

DNDPartners

Community

Partners

Access, Use,Uptake Issues

Marketing,Promotion

When is the Survey Conducted?

Two-week period, chosen by centre, during last fiscal quarter

Centres should maximize legitimate participation during evaluation period

Who is the Survey Data About?

The survey is collecting data that reflects the experience of MFSP participants as defined in MFSP: Parameters for Practice (page 26/27)

Who can Complete the Survey?

One person per family,18 years or over Normally this would be an immediate member of that family and knowledgeable about the entire family experienceIf not an immediate family member then they must ensure that they are responding based on that families experience.Paid staff are not eligible even if participating in services (volunteers are)

To what Extent Should we Assist People to Complete the Survey?

Aim is to help participants with literacy and language issues, disabilities, etc., to complete the survey

Assistance should be provided in a manner that responses are not influenced

Volunteers and other participants are well placed to provide assistance

Can we Combine the Survey with Other Evaluation Activities?

To ensure consistency across C/MFRCs and to maintain the integrity of the process, we ask that no other formal evaluation activities be undertaken during the two-week period

How do we Encourage People to Respond?

Explain purpose and intended use of the survey– Information provided by participants is

valuable Indicate that participation is voluntary but

encouraged Important that staff member presenting the

survey understands purpose and conveys that the process is important

Assure and practice confidentiality

How do we Obtain Quality Responses?

Create a place and time where respondents have time to think and reflect when completing survey

Emphasize that the resulting information is used and thus responses merit careful consideration

Communicate clearly that respondent will not be judged– Open, honest reflection of experiences and

ideas is sought

How do we Assure Confidentiality?

Respondents do not provide their name They seal their completed survey into an

envelope Envelope is placed into a sealed collection box

that will be shipped to DMFS When surveys are analyzed it will be in

aggregate—no individual responses will be reported.

Creating a Script

Break into 3 groups– Write a script for

presenting the survey

– Identify possible questions, issues

– Pick two role players

What Resources will be Available to Help us?

Database to manage,analyze and report data Guidance to survey items PowerPoint for training staff and volunteers

(use as desired) Poster, flyer, short item for newsletter to

publicize upcoming data collection (use as desired)

All materials required to administer the survey (surveys, envelopes, collection boxes)

Constructive Use of Information

Results are viewed as feedbackLearning and future improvement are the

objectivesDevelopment and improvement not

judgment