Capacity-Building Workshop 2013 GEAR UP Evaluation Page 1 GEAR UP Evaluation 101 NCCEP/GEAR UP Capacity-Building Workshop Caesars Palace Las Vegas February

Capacity-Building Workshop 2013 GEAR UP Evaluation • Page 1

GEAR UP Evaluation 101NCCEP/GEAR UP Capacity-Building WorkshopCaesars Palace • Las Vegas • February 4, 2013

Chrissy Y. Tillery • NCCEP • Director of Evaluation

Capacity-Building Workshop 2013


National GEAR UP Objectives

National Objective 1: Increase the academic performance and preparation for postsecondary education for GEAR UP students.

National Objective 2: Increase the rate of high school graduation and participation in postsecondary education for GEAR UP students.

National Objective 3: Increase GEAR UP students’ and their families’ knowledge of postsecondary education options, preparation and financing.


Evaluation Terminology

Qualitative Analyses

• Analysis that involves descriptions and narrative; data is observed.

• Analysis can focus on different types of qualitative analyses including interpretive and narrative, critical theory, participatory action research, phenomenology, etc.

Some examples include: Focus groups Case studies Interviews Ethnography



Quantitative Analyses

• Analysis that involves numbers/inferential statistics; data is measured for growth or significance.

• Embedding quantitative analysis into specific research studies within the overall evaluation is a way to measure more specific outcomes.

Some examples include: Descriptive Statistics

Frequencies, Averages, Percentages t-test ANOVA Regression Propensity Score Matching



Formative Evaluation

• Evaluation conducted and reported on an ongoing basis throughout the project to continuously assess the project.

• Provides program staff with knowledge of how the quality and impact of project activities can be improved.

• Allows for ongoing data-driven decisions to be made.



Summative Evaluation

• Evaluation conducted at the conclusion of the project to assess the overall impact of the project in terms of meeting goals and utilizing efficient resources.

• Used to report final program outcomes.



National GEAR UP Objective• National Objective 1: Increase the academic performance and preparation for postsecondary

education for GEAR UP students.• National Objective 2: Increase the rate of high school graduation and participation in

postsecondary education for GEAR UP students.• National Objective 3: Increase GEAR UP students’ and their families’ knowledge of postsecondary

education options, preparation and financing.

Project Objective – GPRA (Government Performance and Results Act)Performance Indicators• Individualized by grant • Each Project Objective should fall under one of the three National GEAR UP Objectives

Performance Measure• Should include the following:

Baseline Data Target Benchmarks Performance Indicators


Types of Data

• Baseline Data/Pre-Intervention Data• Data collected on students in target schools prior to GEAR UP

intervention

• Intervention Data• Data collected on students in target schools receiving the GEAR UP

intervention

• Post-Intervention Data• Data collected on students in target schools after the GEAR UP

intervention


A Model for Program Evaluation

Continuous

Data Collection

Formative Data Analyses

Program Implementation and RevisionsPolicy Recommendations

Summative Data Analyses


Data Collection Partners

• State Education Agency• Local Education Agencies• University System• Community College System• Private/Independent Colleges and Universities• State Education Assistance Authority• Business Partners• Standardized Testing Agencies – ACT/College Board• National Student Clearinghouse


Evaluation 101: Worksheet 1


Characteristics of Effective Data Collection

• A relational database that is linked by a unique identifier.

• A data system that defines all variables consistently allowing for comparisons.

• A data system that allows for customization related to grant activities.

• A data system that allows for formative and summative evaluation and longitudinal data tracking.

• A data system compliant with FERPA regulations.


Levels of Data Collection

Student Level Data

School Level Data

State Level Data

National Data


Student Level Data

• GEAR UP Student Services• GEAR UP Parent/Family Services• GEAR UP Professional Development services• Student level demographic data• Student level attendance and discipline data• Student level academic data including GPA, state assessment scores, and

course data• Student level dropout and promotion data• Standardized assessment data • Survey data• FAFSA data• National Student Clearinghouse data for enrollment, persistence, and

graduation• Postsecondary data, i.e., remediation data, etc.

*Link data using a unique identifier.


School Level Data

• Percentage of students receiving free and reduced-price lunch

• Percentage of advanced college preparatory courses• Cohort graduation rate• Average daily attendance• Percentage of fully licensed teachers• Percentage of highly qualified teachers• Teacher turnover rate• Percentage of GEAR UP dollars spent in relation to how much

each school was allocated• College Going Culture Data


Evaluation 101: Worksheet 2


Setting Up Your Data

Setting Up Your Data

Non-Technical

Build Relationships

Define Legal Agreements (MOA)

Define Data ElementsTest & Validate Data

Train Staff & Document

Technical

Data System

Linking Tables of Data

Web Interface

Data Entry

Data Loading

Reporting


Data Exchange Considerations

Define file layouts• Various layout options: CSV, XML, etc.• Clearly define the file layout.• Insist on precision from data provider, i.e. requires no manual manipulation on

your end.• Insist on consistency across data feeds, i.e. the file layout does not change.• Ensure clarity in communication.

Define data exchange protocol• Secure FTP,• Direct access to partner’s system to extract data, or• Secure Website, etc.

*Define data change process, i.e., how will changes to data outline be addressed.


Data Inputs and Outputs


Legal Considerations

• Guidance from Legal Counsel• Institutional Review Board (IRB) review• Family Educational Rights and Privacy Act (FERPA)• Confidentiality Agreements• Confidentiality Agreements for GEAR UP Personnel

(GEAR UP staff, Coordinators, etc.)• Confidentiality Agreements for External Consultants

(Consultants, External Evaluators, etc.)


Security Considerations

• Encryption: Make sure steps are taken to encrypt sensitive data elements.

• Efficiency: Monitor databases to ensure data are cleaned and linked.

• Security: Keep the number of users with direct database access to a minimum.

Have users sign a Confidentiality Agreement.

• Disaster Recovery: Make sure your databases are being backed up nightly and

that a clear plan for restoration and recovery is outlined.

• Understand now how long you intend to store data and put measures in place

to ensure that can happen.


National Student Clearinghouse

Postsecondary Data Tracking• StudentTracker for High Schools answers the following questions:

Which of your high school graduates enrolled in college? Where did they enroll? Did they enroll where they applied? Was it their first choice? Did they graduate after six years?

• The National Student Clearinghouse’s database is the only nationwide collection of collegiate enrollment and degree data. These are actual student records provided to the Clearinghouse every 30-45 days by our more than 3,300 participating postsecondary institutions, which enroll over 92% of all U.S. higher education students.

• After StudentTracker matches your records against their database, you’ll receive a comprehensive report containing the information you need to better assess the college attendance, persistence and achievement of your graduates.

• See: http://www.studentclearinghouse.org/


National Student Clearinghouse

Interpreting National Student Clearinghouse Data and setting up files with a unique identifier.


Internal and External Evaluation

GEAR UP must have “implementation of a mechanism to continuously assess progress toward achieving objectives and outcomes, and to obtain feedback on program services and provisions that may need to be altered.”

Internal Evaluator(s): Important to continuously assess the program. Important to have a complete understanding and connection to the program. Important as a trainer for GEAR UP Coordinators and staff in the schools. Important to continuously manage the data for data integrity. Important for day-to-day oversight of evaluation activities.

External Evaluator(s): Important to assess the program from an outside perspective. Important to conduct parallel or independent analysis separate from internal

evaluator(s) for integrity of results. Important that they have knowledge of one or more of the following: (1) GEAR

UP; (2) long term program evaluation; (3) best practices in research methodologies for accurate analysis; and (4) longitudinal analysis.


Evaluation Points to Consider

Research design should match and be appropriate for data collection and analysis.

Evaluation framework should be built around already known local, state, and national data on college-access.

Use prior GEAR UP data to build upon what was successful or what could be strengthened.

Embedded research projects within the overall evaluation can strengthen your proposal and program outcomes.


Evaluation Resources

The Program Evaluations Standards: A Guide for Evaluators and Evaluation Users (3rd Edition) published by the Joint Committee on Standards for Educational Evaluation (2011)

The Institute for Educational Sciences (IES) Practice Guides

The What Works Clearinghouse http://ies.ed.gov/ncee/wwc/

American Educational Research Association (AERA) http://www.aera.net/

American Evaluation Association (AEA) http://eval.org/

http://ies.ed.gov/ncee/wwc/

http://www.aera.net/

http://eval.org/


Capacity-Building Workshop 2013

Thank you for attending the

For additional information regarding the Evaluation 101 session, please contact Chrissy Tillery at 202-530-1135, extension 108 or

[email protected]

NCCEP/GEAR UPCapacity-Building Workshop

Documents

Capacity-Building Workshop 2013 GEAR UP Evaluation Page 1 GEAR UP Evaluation 101 NCCEP/GEAR UP Capacity-Building Workshop Caesars Palace Las Vegas February