Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC)

Leukemia & Lymphoma Society (LLS) Information

Resource Center (IRC)Planning for a Service Program Evaluation - Case

StudyPublic Administration 522

Presented by Ervina Castillo-Newton, John Toadlena and Scott

Atole

October 16, 2012

Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC)

“World’s largest voluntary health organization dedicated to funding blood cancer research, education, and patient services”

Private non-profit service organization Funded by individual and corporate contributions

68 chapters: U.S and CanadaMasters level Information specialists responding to 80,000 inquiries on leukemia, lymphoma, myeloma and other blood cancer patients, etc. Provide information, guidance & support (phone based)

Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation

Background on Service Program Case Study

Evaluation of Information Resource Center(IRC) Email – initial contact Program and academic interest & expertise match Successful response to RFP Initial discussions including questions identified in RFP response.

Additional details expected in the planning process RFP: Pilot a patient navigation model for IRC follow up services

Navigate health care systemsOvercome barriers

Evaluation of service enhancement-scheduled phone call follow up to select callers. Need identified by program staff

LLS-service oriented and committed to the evaluation processEvaluate new programs and ongoing evaluations Internal funding for this evaluation



Initial Planning: 3 months (email to launch) Report provided 6 months later LLS moved to fund a second/larger evaluation on Patient navigation

EPIC model (steps before the pilot study)



Understanding the Political & Organizational EnvironmentPre-project-consulted various sources

Internal/external people Internet/LLS website LLS Promotional items/collaterals RFP LLS Mission Statement LLS Organizational structure (background materials/site visit)

Commitment to organizational cause. Political environment associated with advocacy and

organizational commitmentFunding-individual/corporate LLS operations (private sector-accountability)How will the evaluation findings be used? Future funding

from LLS Board of trustees. Internally competitiveLocation of IRD within LLS structure. department


Assess the Context

Understanding the Political & Organizational EnvironmentLLS values program evaluation

Objective data to support the IRC program enhancementIRC – immediate program context

7 full time information specialists 80,000 phone calls annually Navigation model – innovative among other national programs Navigation to identified subgroup (2 specialists)

National organization 68 chapters – IRC link to local chapters Need for cancer information (consumer friendly) Potential for collaboration “community of cancer information service

professional”


Assess the Context

Define the Relationship Between Evaluator & Sponsor Internal/External Evaluator and Funding

LLS External Evaluator/Internal fundingFindings were reported to the LLS Board of TrusteesPrimary Point of Contact – LLS Vice President or Patient Services & Disease Programs (Includes IRC)

Advantages/Disadvantages-resistance to int/ext pressures, produce positive findings, objectivity, credibility, timelines(learning/building relationship), potential biases

Clarify roles and the relationship earlyEvaluator and Sponsor rolesFunding arrangements

LLS –Participatory Relationship in planning and implementationLLS – 2 ManagersIRC – 1 Supervisor (Ongoing Liaison) and 1 staff member

Ongoing Education – ResponsibilitiesLLS Staff – Provide feedbackEvaluator-Drafts, Technical assistance (research process), Assess stakeholder knowledge

Evaluator skillRelevant experience, experience and interestQualitative vs. Quantitative


Assess the Context

Determine the Level of the Evaluation Leukemia & Lymphoma Society (LLS) is local Information Resource Center (IRC) is national. Seek to create the connection with local chapters. LLS Target population

Newly DiagnosedThose seeking information on clinical trialsPatients residing in Wyoming

Reasoning for Target Population selection (Where will this enhancement be most helpful?):

Limited LLS resources to offer the program to all callersNavigation approach –reduction in health disparities to vulnerable populationsWyoming – no local chapters

Outcome vs. Process EvaluationPrimary focus on outcomesIRC program enhancement – include and document the process


Assess the context

Specify Evaluation UseLLS-findings will be used for program sustainability and continued and amount of future funding. Evaluations used to make decisions. Important to understand the implication of negative findingsWill this program be duplicated? Will the literature/results be made available nationally?

Validate PerspectivesCommunications with nearly every level of LLS/IRCWelcome evaluationSupportive of the projectImportant to understand that it is a program evaluation


Gather Reconnaissance

Potential stakeholders for a service program include: Two main reasons why the evaluator must engage stakeholders in

evaluation planning. Identify and Invite Stakeholders.

Establish rapport. Participants involved in the evaluation planning. “Sponsor” role.

Define Stakeholder Roles and Structure for Their Input. Importance of engaging the few stakeholder groups. Survey design.

John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation

Engage Stakeholders

Establish Group Processes for Ongoing Stakeholder Involvement. The communication process.

Methods used.Evaluation plan timeline.Preliminary reports.


Engage Stakeholders

Establishment of the stakeholder. Become immersed in the nature of the program to be evaluated. “Framework for Program Evaluation in Public Health”. Understanding the IRC. Conceptualize Program Theory or Rationale.

Usefulness of logic models and conceptual models. Evaluation Planning Matrix (EPM). (Holden and Zimmerman, pg.78-79)

Lists the evaluation questions considered. Short-term, Intermediate, or Long-term outcomes and identifies relevant data

sources corresponding to each question. Outcome surveys and questions.


Describe the Program

Focusing the evaluation. The planning process. Study design team. Preliminary evaluation questions. Theoretical frameworks. Ensure Feasibility.

The focus on feasibility. Control or comparison group.

Assess Potential Data Collection Burden. Data collection burden an issue on two levels.

How much surveying could the participants tolerate. What kinds of data collection operation could IRC take on. Sample size.Draw conclusions from the results.


Focus the Evaluation

Service Programs are unique Start with the Mission Statement

Size and scope of the activities Evaluation

Attitudes Stakeholders’ knowledge

Stakeholders Expertise & knowledge Minimizes resistance

Planning Process Dissemination Goals This case – positive and knowledgeable

Ervina Castillo-Newton UNM-WS PADM 522 Planning for a Service Program Evaluation

Lessons Learned

Comparison GroupBased on comparison of callers

Standard Service through IRC Enroll in intervention

Information Specialists Qualified Trained Experienced

Outcomes

Ervina Castillo-Newton UNM-WS PADM 522 Planning for a Service Program Evaluation

Conclusion

Documents

Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC)