Embed Size (px)
Citation preview
AAA Monitoring Framework
Elizabeth O’Neill, Program EvaluatorState of Oregon DHS APD State Unit on Aging
March 2014
Overview
• Purpose of initiating monitoring efforts
• State assumptions
• Central questions
• Evaluation vs. monitoring
• Monitoring framework adapted from GAO
• Monitoring next steps: instructions & effort
Purpose of Monitoring• Create consistent framework for
quality assurance with local variation
• Share best practices statewide
• Identify & correct areas of weakness
• Comply with federal and state regulations; satisfy previous and future audits
• Strengthen service delivery
Assumptions
Assumptions• SUA assumes AAAs have in-depth, expert knowledge of
the national and state requirements.
• SUA assumes that AAAs are doing regular, thorough monitoring of their program and operational systems.
• SUA assumes AAAs are operating under the practices they believe to be the most efficient and equitable.
• SUA believes that AAA are actively engaged in sharing quality assurance practices, though this process could be substantiated.
• SUA believes that we should be able to share program quality information with legislators and other stakeholders in a more substantive way.
Assumptions continued• SUA believes that standards may be
strengthened and efficiencies gleaned through ongoing monitoring efforts.
• SUA believes the statewide aging network needs a more systemic monitoring system.
• SUA believes that a self-reported monitoring system will produce factual data and honest self-reflection.
• SUA believes we can leverage our small capacity through AAA self monitoring and make the process more efficient for AAAs.
Monitoring Intent
Program Standards
o Fiscal
o Business & Operations
Central Questions
• How are specific AAAs meeting program standards?
• What are local and statewide trends?
• What are the best practices?
• How do AAAs propose continuing areas that are working well and fixing areas that are not meeting standards?
• What can the SUA do to better support AAAs in meeting standards?
• Do the reviewed programs meet national and state standards?
Understanding Evaluation vs. Monitoring
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding
Finding• Finding is the
compilation of collecting specific data, analyzing it and determining the condition.
• A finding is dependent on evaluation objectives.
• A finding can be positive or negative in this context.
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding
Criteria “What should be?”
• Provides information so that the user will be able to determine what is required, desired, or expected from the program or operation.
• Criteria is often program standards or rules.
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding
Condition “What is?”
• Condition provides evidence on what is found regarding the actual situation.
• Reporting the scope or extent of the condition allows the user to gain an accurate perspective about program status.
• The monitoring process will reveal the information about the central questions.
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding
Cause “Why the condition happened?”
• Cause provides persuasive evidence on the factor(s) responsible for the difference between condition & criteria.
• Cause explains why the difference between the ideal & actual circumstances exist.
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding
Effect Difference between “what is” &
“what should be?”• Effect provides a clear, logical link to establish the
impact of the difference between condition & criteria.
• Effect is easier to understand when it is stated clearly, concisely, and if possible, quantifiable.
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding
What can be perpetuated to replicate
success?
What actions are needed to correct the cause?
Recommendations address how to align criteria and condition and should directly address the cause.
Recommendations
ConditionWhat is
CriteriaWhat should be
CauseWhy the condition happened
EffectWhat is the
impact
RecommendationsWhat will address the cause
Elements of a Finding: SUA’s Model State or Federal program guidelines
& contractual requirements
Results of monitoring efforts as demonstrated in the assessments
Reflection from AAA
Reflection from AAA
Designed by AAAFollowed up and celebrated by SUA
• Also contains monitoring responsibilities for SUA and AAAs
• Ramifications for lack of compliance
• Links to program standards for OPI, Family Caregiver Support Program and Health Promotion.
Assessment tools for OPI, FCSP & Health Promotion due to SUA by 5/23/2014.
Monitoring Instructions
Monitoring Effort
• All monitoring efforts will be done by the reporting AAA for 2013-14.
• A small number of programs will pilot peer monitoring in 2014-15 where AAAs will partner and conduct an assessment for one another.
• Both self-monitoring and peer-monitoring will be verified by SUA through abbreviated on-site reviews.
Location of Monitoring Materials
SUA website Area Agency on Aging business AAA Self Monitoringhttp://www.oregon.gov/dhs/spwpd/pages/sua/info-aaa.aspx
Resources• GAO Yellow Book http://www.gao.gov/yellowbook/overview
• GAO Designing Evaluations http://www.gao.gov/products/GAO-12-208G
• GAO Performance Management and Evaluation http://www.gao.gov/assets/80/77277.pdf
• Institute of Internal Auditors http://www.theiia.org/intAuditor/back-to-basics/2013/report-audit-findings/
• AARP Public Policy Institute’s focus on Livable Communities http://www.aarp.org/research/ppi/liv-com/
• Program Evaluation and Practice: A Comprehensive Guide; Mertens and Wilson
• Improving Performance in Service Organizations: How to Implement a Lean Transformation; Miller, Bogatova, Carnohan
