Janet BarresiSTATE SUPERINTENDENT OF PUBLIC INSTRUCTIONOKLAHOMA STATE DEPARTMENT OF EDUCATION

MEMORANDUM TO: District Superintendents FROM: Janet Barresi, State Superintendent of Public InstructionDATE: Tuesday, September 9, 2014 SUBJECT: SY2014-2015, Title II, Part B, Mathematics and Science Partnerships Program Enclosed is the SY2014-2015, Title II, Part B, Mathematics and Science Partnerships (MSP) Program Request for Proposals. Eligible applicants for this grant consist of partnerships that include a mathematics, science, or engineering department of an Oklahoma institution of higher education and a high-need K-12 local educational agency (LEA). A high-need LEA must be the lead district and thus will carry the fiscal responsibilities for this grant. This includes the use of Indirect Cost (IDC) in which the grant must use the lead high-need LEA’s IDC rate. Eligible partners may include other mathematics, science, engineering, or teacher training departments of institutions of higher education, additional school districts (which may or may not be high need), public charter schools, public or private elementary schools, or secondary schools or consortia of such schools, businesses, or nonprofit or for-profit organizations of demonstrated effectiveness in improving the quality of mathematics and science teachers. The term “high-need school district” is defined for this project as the following. Please remember only one of the following requirements must be met to qualify as the lead LEA.

A. has at least 40 percent of the children are from families with incomes below the poverty line based on the LEA’s Free and Reduced Lunch Count; OR

B. has 20 percent poverty determined by the census; OR

C. has a site(s) designated as school improvement for the 2014 school year*; OR

D. has any mathematics and/or science classes not taught by highly qualified teachers.

Grant Period: The grant period will be February 1, 2015 to June 30, 2016. Grant Award: Five to ten grants ranging from $75,000 to $250,000 will be awarded. Announcement of Grant Awards: Grant awards are tentatively scheduled to be announced in January 2015. Letter of Intent: A non-binding letter of intent should be submitted to Jeff Downs (Jeff.Downs@sde.ok.gov) by Friday, September 26, 2014.

1

JANET BARRESISTATE SUPERINTENDENT OF PUBLIC INSTRUCTIONOKLAHOMA STATE DEPARTMENT OF EDUCATION

Access Application Documents: The 2014-2015 MSP Request for Proposal, Appendices, and Proposal Framework are available online at http://sde.ok.gov/stem.

Application and Submission Deadline: Applications must be physically received no later than 5:00 p.m., Friday, December 5, 2014. Send original copy of the full proposal along with jump drive or CD-ROM one complete proposal file in Microsoft Word (read-only) or PDF format. Incomplete, late, or incorrectly formatted proposals will not be scored or considered for funding.

• Applicants are encouraged to use express, certified, or registered mail.• Faxed or emailed proposals will not be accepted.• Mail proposals to:

Jeff Downs, M. Ed.Executive Director of STEM

2500 N. Lincoln Blvd., Suite 315OKC, OK 73105-4599

Contact Information: For questions regarding this grant, please contact the following staff:

Jeff DownsExecutive Director of STEM Office of Instruction (405)521-2107 | Jeff.Downs@sde.ok.gov

Dr. Gloria BayouthExecutive Director for Titles I, II, VI & X Federal Programs (405)522-3249 | Gloria.Bayouth@sde.ok.gov

Tiffany NeillDirector of Science EducationOffice of Instruction (405) 522-3524 | Tiffany.Neill@sde.ok.gov

Levi PatrickDirector of Secondary Math Education Office of Instruction (405) 522-3525 | Levi.Patrick@sde.ok.gov

Jennifer LambDirector of Elementary Math Education Office of Instruction (405) 522-3522 | Jennifer.Lamb@sde.ok.gov

2

JANET BARRESISTATE SUPERINTENDENT OF PUBLIC INSTRUCTIONOKLAHOMA STATE DEPARTMENT OF EDUCATION

Announcement Webinar: Announcing SY2014-2015 Math and Science Partnership Grant Cycle

Tuesday, September 9, 2014 10:00 a.m. - 11:00 a.m.WebEx - https://oksdetraining.webex.com

Presenters: Jeff Downs, Executive Director of STEM, Office of Instruction

Purpose: To introduce the Mathematics and Science Partnerships (MSP) Program and to take inquiries specifically concerning the 2014-2015 Mathematics and Science Partnerships (MSP) Program Request for Proposals (RFP).

Participants: Superintendents, principals, federal program directors, and anyone interested in writing for the MSP grant.

Agenda: 1. Welcome and Introduction 2. Basic Overview of the New Request for Proposal structure 3. Questions and Answers

Technical Assistance Webinars: Guidance and Support for SY2014-2015 Math and Science Partnership Grant Cycle

Tuesday, September 22, 2014 AND Tuesday, September 23, 201410:00 a.m. - 11:00 a.m.WebEx - https://oksdetraining.webex.com

Presenters: Jeff Downs, Executive Director of STEM, Office of Instruction

Purpose: To provide technical assistance regarding new elements of the Oklahoma MSP, partnerships, needs assessments, work plans, evaluations, and budgets in a two-part webinar.

These sessions will be taped and archived. Links to taped sessions can be obtained by visiting the OSDE website STEM page: http://ok.gov/sde/stem

3

Oklahoma State Department of Education

Mathematics and Science Partnership (MSP) Program

Request for Proposals (RFP)

2014-2015

Title II Part BNo Child Left Behind Act of 2001

Public Law 107-110

RFP Published: Tuesday, September 9, 2014

Letter of Intent to Apply Due: Friday, September 26, 2014 by 5:00 p.m.

Proposals Due: Friday, December 5 , 2014 by 5:00 p.m.

Grant Award Notification: January 2015

Grant Period: February 1, 2015 - June 30, 2016

4

Oklahoma Mathematics and Science Partnership (MSP) Program Executive SummaryPurpose: The purpose of the Oklahoma Mathematics and Science Partnership (MSP) Program is to improve the content knowledge and pedagogical practices of cohort groups of mathematics and/or science teachers of grades K-5, 6-8, and/or 9-12 in order to increase the achievement of their students. These improvement efforts are designed, implemented, and evaluated by strong partnerships between college and university faculty, high-need school districts, and other qualifying partners. Eligibility: An eligible partnership is one that demonstrates deep and mutual engagement between

(a) one or more school districts, at least one of which must meet high-need criteria (see page 8); and(b) science, technology, engineering, and/or mathematics (STEM) faculty at an accredited 2 or 4 year college or university.

Programs may also include additional accredited colleges or universities as well as faculty from the department or division responsible for the preparation of teachers (typically the college of education), businesses, and non-profit and for-profit organizations with proven effectiveness in providing professional development to teachers of mathematics and science.

Priorities of the OSDE: In addition to the purpose and partnership eligibility descriptions listed above, the Oklahoma State Department of Education (OSDE) places funding priority on partnerships that

(a) recruit and serve teacher cohort groups from schools with the greatest academic or instructional need;

(b) show evidence of ways in which building-level administrators will meaningfully participate in the partnership’s professional learning experience; and

(c) create innovative approaches for instruction and/or partnerships. Estimated Amount to be Awarded: $1,700,000

Anticipated Number of Awards: 5-10 Award Distribution: The OSDE intends to fund MSP projects equitably and to distribute the projects across the state to the extent that submitted, qualified proposals allow. Duration of Grants: Grants funds will be available February 1, 2015 and shall be completely expended no later than June 30, 2016. Fiscal Agents: Fiscal responsibility for the grant must rest with the lead district partner. If an eligible partner is determined by the entirety of the partnership to have the greater capacity to serve in that role, requests for modification may be submitted directly to Jeff Downs (Jeff.Downs@sde.ok.gov) and will be reviewed by the full OSDE MSP Team on a case-by-case basis. Requirements of Awarded Applicants: If awarded MSP funds, all awardees will be required to submit budgets through OSDE’s Online Grants Management System. In addition, each project must be represented at the U.S. Department of Education’s regional meeting each year. An online Annual Performance Report must be submitted to OSDE by Friday, September 18, 2016 and the State will then review and submit that report to the U.S. Department of Education. All awarded projects will receive monitoring for both programmatic and fiscal compliance. Projects should expect one or more site visits each year from OSDE staff and numerous visits from the external evaluator. Further guidance on these requirements is provided during the Technical Assistance Webinars. Intent to Apply: Applicants should submit a non-binding notice of Intent to Apply via email to Jeff Downs (Jeff.Downs@sde.ok.gov), the Executive Director of STEM, by Friday, September 26, 2014. These intention letters should list the possible partnering LEA(s) and institute(s) of higher education

5

in addition to the subject and grade levels the partnership intends to work with. This helps OSDE in the creation of the review panel and also allows the OSDE MSP Team to notify institutions if they are involved in multiple applications thus competing against themselves.

For this competition, an LEA may submit only one proposal as the lead partner of an MSP project. That LEA may be included as a secondary partner on proposals by other partnerships that do not seek to provide professional learning opportunities in the grade levels and content area(s) already provided for by said LEA.

Proposal Delivery: Partnerships must send one original of the complete proposal along with a CD-ROM or jump drive containing one complete proposal file in Microsoft Word (read-only) or PDF format.

● All proposals must be physically received by the OSDE by 5:00 p.m. on Friday, December 5, 2014. Incomplete, late, or incorrectly formatted proposals will not be scored or considered for funding.

● Applicants are encouraged to use express, certified, or registered mail.● Faxed or emailed proposals will not be accepted.● Mail proposals to:

Jeff Downs, M. Ed.Executive Director of STEM

2500 N. Lincoln Blvd., Suite 315OKC, OK 73105

Review and Notification of Awards: It is the intention of the OSDE MSP Team to convene an expert review panel in December and to present funding recommendations to the Oklahoma State Board of Education at its January 2015 meeting. Therefore, the OSDE MSP Team anticipates announcing award decisions to partnerships in January 2015, with grants officially beginning February 1, 2015.

6

Table of ContentsProposal Section Page

NumbersOklahoma MSP Program Executive Summary 5Section 1: MSP Program Overview 8

● Federal Program Overview● Oklahoma Program Overview

Section 2: Oklahoma MSP Program Requirements and Administration Information 10

● High-Need Criteria● Eligible Partnerships● Sustainable Partnership Roles● Partner Organization Proposal Limit● Fiscal Responsibility● Usage of Funds● Allowable Expenditures● Budget Design Considerations● Maximum Eligible Costs● Ineligible Costs● Anticipated Grant Competition Timeline

Section 3: Oklahoma MSP Program Key Features 18

● Partnership Qualities● Project Work Plan Elements● Project Evaluation and Accountability Plan● Providing Services to Eligible Nonpublic Schools● FERPA

Section 4: Oklahoma MSP Proposal Format and Submission 23

● Format Requirements● Proposal Delivery

Section 5: Oklahoma MSP Program Proposal Preparation 24

● Grant Application Cover Page● Partner List● Statement of Assurances● Repeat Applicant Project Abstract● Proposal Abstract● Results of Needs Assessment● Work Plan: Goals/Objectives, Action Plan, Management Plan● Evaluation and Accountability Plan● Budget and Budget Narrative● Appendix

Section 6: Oklahoma MSP Program Review and Award Process 28

● Review Process● Review Criteria/Rubrics● Notification of Award

Appendices 30

● Possible Resources for Proposal Preparation (p.30)● Guide for Summarizing MSP Evaluation Designs and Results (p.31)● Scoring Rubric for Applicants Not Receiving Funds in the Previous Cycle (p.47)● Scoring Rubric for Repeat Applicants (p.51)● Professional Development Standards for the TEACHER (p.56)● Definitions (p.59)

7

Section 1: Mathematics and Science Partnership (MSP) Program OverviewFederal MSP Program OverviewThe Mathematics and Science Partnership (MSP) Program is funded under Title II, Part B of the No Child Left Behind Act of 2001. Its purpose is to improve the content knowledge and teaching skills of mathematics and/or science teachers in order to increase the achievement of their students.

To be eligible, a partnership must include, at a minimum:● an engineering, mathematics, or science department of an Institute of Higher Education (IHE);

and● a high-need Local Education Agency (LEA).

A partnership may include:● another engineering, mathematics, science or teacher training department of an IHE;● additional LEAs, public charter schools, public or private elementary schools or secondary

schools, or a consortium of such schools;● a business; or● a nonprofit or for-profit organization of demonstrated effectiveness in improving the quality of

mathematics and science teachers.

Partnerships assume responsibility for designing, implementing, and evaluating professional learning programs that affect deep, lasting improvement in mathematics and science education by:

a) establishing and operating intensive mathematics and science professional learning experiences for teachers with ongoing follow-up training and support that improves their content knowledge and instructional practice; and

b) using scientifically-based researched teaching methods to promote strong teaching skills for mathematics and science teachers.

AUTHORIZED ACTIVITIES - An eligible partnership shall use funds provided under this part for one or more of the following activities related to elementary schools or secondary schools:

1. Creating opportunities for enhanced and ongoing professional development of mathematics and science teachers that improves the subject matter knowledge of such teachers.

2. Promoting strong teaching skills for mathematics and science teachers and teacher educators, including integrating reliable scientifically based research teaching methods and technology-based teaching methods into the curriculum.

3. Establishing and operating mathematics and science summer workshops or institutes, including follow up training, for elementary school and secondary school mathematics and science teachers that —

a. shall —i. directly relate to the curriculum and academic areas in which the teacher provides

instruction, and focus only secondarily on pedagogy;ii. enhance the ability of the teacher to understand and use the challenging State

academic content standards for mathematics and science and to select appropriate curricula; and

iii. train teachers to use curricula that are —1. based on scientific research;2. aligned with challenging State academic content standards; and3. object-centered, experiment-oriented, and concept- and content-based;

and b. may include —

i. programs that provide teachers and prospective teachers with opportunities to work under the guidance of experienced teachers and college faculty;

ii. instruction in the use of data and assessments to inform and instruct classroom practice; and

8

iii. professional development activities, including supplemental and follow-up activities, such as curriculum alignment, distance learning, and activities that train teachers to utilize technology in the classroom;

4. Recruiting mathematics, engineering, and science majors to teaching through the use of —a. stipends provided to mathematics and science teachers for certification through

alternative routes; andb. scholarships for teachers to pursue advanced coursework in mathematics, engineering, or

science;5. Developing or redesigning more rigorous mathematics and science curricula that are aligned with

challenging State and local academic content standards and with the standards expected for postsecondary study in mathematics and science.

6. Establishing distance learning programs for mathematics and science teachers using curricula that are innovative, content-based, and based on scientifically based research that is current as of the date of the program involved.

7. Designing programs to prepare a mathematics or science teacher at a school to provide professional development to other mathematics or science teachers at the school and to assist beginning and other teachers at the school, including (if applicable) a mechanism to integrate the teacher's experiences from a summer workshop or institute into the provision of professional development and assistance.

8. Establishing and operating programs to bring mathematics and science teachers into contact with working scientists, mathematicians, and engineers, to expand such teachers' subject matter knowledge of and research in science and mathematics.

Oklahoma MSP Program Description and GoalsThe Oklahoma State Department of Education (OSDE) is responsible for administering the MSP program and is authorized to award approximately $1,700,000 in competitive grants as of February 1, 2015. Grants will be awarded to eligible partnerships for a period of one year, subject to (a) compliance with program requirements, (b) demonstration of effectiveness, and (c) availability of federal funding. School districts are concentrating their efforts on adjusting instruction to prepare greater numbers of students for high achievement in mathematics and science. The Oklahoma MSP Program strives to support these efforts to improve grades K-12 mathematics and science teacher quality by immersing teacher cohort groups in sustained, creative, and strategic professional learning that extends beyond commonplace approaches to improve mathematics and science achievement. This cohort-based approach will enable teachers to see themselves as integral members of a professional community linked with others devoted to learning and practice. The Oklahoma MSP Program seeks to improve the content knowledge and ability to analyze student thinking of mathematics and science teachers in grades K-12. More specifically, the program strives to meet the following goals:

● increase the capacity of grades K-5, 6-8, and/or 9-12 mathematics and/or science teachers to improve student achievement, particularly in schools with the greatest instructional and academic need;

● increase the number of grades K-5, 6-8, and/or 9-12 mathematics and science teachers who participate in content-based professional learning and who are prepared to teach challenging courses and curricula.

The OSDE anticipates funding 5-10 projects showing the potential to accomplish these goals and will distribute the awards to projects across the state to the extent that submitted, qualified proposals allow.

9

Section 2: Oklahoma MSP Program Requirements and Administration InformationTo increase the likelihood of reaching these goals, the OSDE has set specific requirements for partnerships in terms of high-need criteria, partnership eligibility, use of funds, allowable expenditures, and the anticipated grant competition timeline. High-Need Criteria

A school district is considered to be high-need by the Oklahoma MSP Program if it meets the following criterion:

● has at least 40 percent of the children are from families with incomes below the poverty line based on the LEA’s Free and Reduced Lunch Count; OR

● has 20 percent poverty determined by the census; OR● has been designated school improvement for the 2014 school year*; OR ● has any mathematics and/or science classes not taught by highly qualified teachers.

(All teachers providing direct instruction in mathematics or science, including special education teachers, need to meet the highly qualified requirements of the No Child Left Behind Act.)*

* If a district qualifies because of school improvement status or because of a percentage of mathematics and/or science classes not taught by highly qualified teachers, the application must target the area that is the qualifying factor.

Eligible Partnerships

Partnership is critical to the success of individual MSP projects. Partnerships eligible to apply for an MSP Program grant must include:

● at least one high-need school district and● the science, engineering, or mathematics department of an accredited 2 or 4 year college

or university in Oklahoma. ○ Requests to partner with universities outside of Oklahoma will be considered on a

case-by-case basis. Send inquiries to Jeff Downs (Jeff.Downs@sde.ok.gov). Partnerships may also include:

● another engineering, mathematics, science or teacher preparation unit of an IHE;● additional LEAs, public charter schools, public or private elementary schools or

secondary schools, or a consortium of such schools;● a business; or● a nonprofit or for-profit organization of demonstrated effectiveness in improving the

quality of mathematics and science teachers.

Sustainable Partnership Roles Partnerships must have a management structure in which each partner is fully represented and engaged, including project leaders from each of the remaining organizations. In addition, it is recommended that one teacher from each participating school/district serve on the management team. This project management team must meet regularly to oversee all phases of the project, including design of the project, recruitment and retention of the teacher cohort group, implementation of the project plan, and collection and analysis of data related to its impact on teaching and learning. Key elements for the Partnerships:

● partners are equal and make collaborative decisions;● roles for scientists and mathematicians are clearly defined;● consistent vision, values, goals and objectives are shared by all partners;● communication is consistent and deliberate;● there are benefits to teachers;

10

● there are benefits to students; and● there are benefits to scientists and mathematicians.

Additionally, the project management team has collective program responsibilities:

● submit the following reports:○ Evidence of Project Management Team meetings;○ 3 Budget summary reports,○ 4 Follow-up professional development reports; ○ Evaluators Report

● submit an annual performance report to the OSDE;● designate participants to the Annual MSP Conference hosted by U.S. Dept. of ED;● participate in Program leadership team work sessions facilitated by the OSDE MSP

Team; and● utilize the PRISM Partnership Rubric as a guide to establish and facilitate stakeholder

partnerships http://prism.mspnet.org/index.cfm/10013

During the grant period, site visits from one of the OSDE MSP Team members should be expected. It is the responsibility of the project director and the project management team to ensure that the OSDE MSP Team is kept current as to when and where the professional learning sessions will take place.

Partner Organization Proposal Limit

For this competition, an organization may submit only one proposal as the lead partner of an MSP project. That organization may be included as a secondary partner on proposals by other partnerships that do not seek to provide professional learning opportunities in the grade levels and content area(s) already provided for by said organization.

Fiscal Responsibilities

The OSDE has determined that fiscal responsibility for the grant must rest with the lead district partner, which must have a Fund 11 established. If an eligible partner is determined by the entirety of the partnership to have the greater capacity to serve in that role, requests for modification may be submitted directly to Jeff Downs (Jeff.Downs@sde.ok.gov) and will be reviewed by the full OSDE MSP Team on a case-by-case basis. Indirect costs may not exceed 8 percent (or the institutions federally negotiated indirect cost rate, whichever is lower) for its role as fiscal agent.

Usage of Funds

A partnership may use MSP Program funds to create opportunities for enhanced and ongoing professional learning for mathematics and/or science teachers of grades K-12 that improves their content knowledge and pedagogical practices and make corresponding instructional decisions by establishing and operating mathematics and/or science intensive experiences and related follow-up training and support that:

● directly relate to the curriculum and content in which the teachers provide instruction yet provide instruction at a level beyond the level of content they are expected to teach to students;

● improve the ability of the teachers to understand and use the State Standards● improve the ability of teachers to integrate and to understand applications of mathematics

and science;● provide instruction and practice in the effective use of content-specific pedagogical

strategies; and● provide instruction in the use of data and assessments to inform mathematics and science

classroom practice.

11

Allowable ExpendituresOklahoma MSP Program funds must be spent exclusively on costs associated with providing high quality professional learning opportunities to mathematics and/or science teachers of grades K-12.

Budget Design ConsiderationsFor any person whose project duties include both administrative and instructional services, create separate budget entries showing the requested amount for each set of services. Describe the grant-related services to be provided, as well as whether or not the person is working outside regular hours and describe each benefit and its percentage when benefits other than FICA are being requested. The applicant must provide a direct link for each cost to the goals and objectives in the project Activity Plan. In general, it is expected that MSP partnerships will spend approximately $45-$55 per teacher per contact hour on the total cost of their MSP Program work. For example, if 50 teachers work for 80 hours at a Funding Rate of $45 to $55 per teacher, the range of expected costs will be $180,000 to $220,000, where Total Cost = # of Teachers x Total Contact Hours x Funding Rate Per Teacher.

For full-time employees working a part or all of their regular work day on the grant, applicants must describe the actual professional development instruction or coaching (instructional salaries) duties to be performed and to whom they are providing the services. Applicants must be sure to include an appropriate cost basis such as the hourly rate and the number of hours worked. For salaries, show the annual salary (if less than 12 months be sure to identify the percentage of time covered by the salary) and the percentage of that salary being paid by the grant. The OSDE will disallow all ineligible costs, as well as costs not supported by the Project Activity Plan. These funds will not be eligible for reallocation. Grant funds must be used to supplement and not supplant existing efforts of the LEA. Federal funds cannot be used to pay for anything that a grant applicant would normally be required to pay for with either local or state funds. This requirement also covers services previously provided by a different person or job title. The exceptions are for activities and services that are not currently provided or statutorily required, and for component(s) of a job or activity that represent an expansion or enhancement of normally provided services.

12

Maximum Eligible CostsOMB Circulars establish spending rules for recipients and sub-recipients of all federal funds. OMB Circulars can be located at http://www.whitehouse.gov/omb/circulars_default.

Stipends: Funds may not be used to augment the total salary or salary rate of faculty/staff members during the period covered by the term of faculty appointment or to reimburse faculty members for consulting or other time in addition to a regular full-time organizational salary covering the same general period of employment. Exceptions may be considered for weekend, evening classes, or for administrative work done as overloads. The names of the Project Director, faculty, and other senior personnel and the estimated number of full-time-equivalent academic-year, summer, or calendar-year person-months for which funding is requested and the total amount of stipends requested per year must be listed. Stipends requested must be consistent with the organization’s regular practices. The budget justification should detail the rates of pay by individual. It is permissible for the Project Director to budget for project management as time required in addressing the specifically named goals and objectives of the project. As with all uses of federal grant funds, the grantee will need to maintain records to document that payment of stipends is reasonable and necessary to the approved project.

Contract Services: Not to exceed $800 per full day for professional services. Consultant expenses should be calculated according to the state regulations governing travel and lodging expenses. (Time and Effort Logs Required.)

Except for the procurement of such items as commercially available supplies, materials, or general support services allowable under the grant, no significant part of the substantive effort under the grant may be contracted or otherwise transferred to another organization without prior authorization.

The intent to enter into such arrangements must be disclosed in the proposal, and a separate budget should be provided for each contract, if already identified, along with a description of the work to be performed. Otherwise, the disclosure should include a clear description of the work to be performed, and the basis for selection of the contractor.

Project Management Professional Development: Project Directors and up to one other project

member are required to attend one MSP Annual Meeting that is conducted by the U.S. Department of Education. MSP funds can be used to support travel expenses. MSP funds should be budgeted for these events. The trips are usually held in Washington, D.C. Travel per diems can be found at http://www.gsa.gov/portal/content/104877.

External Evaluation Services: No more than 8 percent of the total award can be used for

external evaluation services. External evaluators should not be affiliated with any of the institutions in the partnership. If conducting a quasi-experimental evaluation design, additional funds may be justified for an evaluator. (See Summary of Guidelines for Allowable Expenses on page 16.)

Teachers’ Compensation: The grant program's maximum allowable contribution to teacher

compensation is $25 per hour and is to be used as the standard rate of pay no matter what the length of the overall day determined.

Tuition: Annual tuition payment (payable to the IHE where the credits will be earned and coded)

for graduate course credits is permissible if the course and participant meet all four of the following criteria:

1. the course is directly related to the MSP participants’ professional development plan;

13

2. the course will lead to the completion of an accredited graduate education program/endorsement;

3. the participant successfully completes the course with a grade of B or better; and4. the tuition for a course is not already provided by the LEA.

Travel: Travel expense reimbursement is limited to the state-approved rate per mile and per diems. Other travel arrangements should be made by the least expensive means available. Travel and its relation to the proposed activities must be specified and itemized by destination and cost. Funds may be requested for field work, attendance at meetings and conferences, and other travel associated with the proposed work, including subsistence. In order to qualify for support, however, attendance at meetings or conferences must be necessary to accomplish proposal objectives, or disseminate its results. Allowance for air travel normally will not exceed the cost of round-trip, economy airfares. Persons traveling under project must travel by US-Flag air carriers, if available. Out-of-state conference travel must be limited to the MSP Annual Conference only.

Materials and Supplies: Funds may be spent on materials and supplies to facilitate the professional learning of teachers. The proposal budget justification should indicate the general types of expendable materials and supplies required. Materials and supplies are defined as tangible personal property, other than equipment, costing less than $5,000, or other lower threshold consistent with the policy established by the proposing organization. Cost estimates must be included for items that represent a substantial amount of the proposed line item cost. Instructional materials can only be purchased for the teacher attending the professional development for the purposes of the program (federal funds may not be used to purchase equipment or instructional materials for the students of the teacher).

Subgranting: The Lead LEA must be aware that subgranting this award is not allowable.

Restricted Indirect Costs: 8 percent is the maximum restricted, indirect cost rate allowed (EDGAR §76.567(c)(2)). The indirect cost rate applies only to direct costs, not the total award amount received. Applicants must use one of the two following indirect cost rates, whichever is lower:

a) 8 percent; orb) the lead LEA’s indirect cost rate.

14

Ineligible Costs:● Costs associated with writing the application;● Equipment (all equipment requested is subject to review and approval by the OSDE);● Full salaries of administrative or clerical personnel1;● Tuition charges and/or university/activity fees already covered in the higher education partners’

salary and fringe;● Capital improvements;● Food; ● Supporting the research of individual scholars or faculty members;● Providing compensation for IHE faculty attending workshops or conferences other than U.S.

Department of Education Mathematics and Science Partnership Conferences;● Supporting travel to out-of-state professional meetings, unless it is demonstrated that attendance at

a meeting will directly and significantly advance a project.● Costs that are not directly related to the educational program and that are unsupported by the

proposal; and● Entertaining

15

1 In most circumstances, salaries of administrative or clerical staff are included as part of indirect costs (also known as Facilities and Administrative Costs (F&A) for Colleges and Universities). Partial salaries of administrative or clerical staff may be requested as direct costs for a project requiring an extensive amount of administrative or clerical support and where these costs can be readily and specifically identified with the project with a high degree of accuracy. The circumstances for requiring direct charging of these services must be clearly described in the budget justification. Such costs, if not clearly justified, may be deleted. See OMB Circular A-21 (2 CFR Part 220) and OMB Circular A-87 (2 CFR Part 225), for examples of where direct charging of administrative salaries may be appropriate.

Summary of Guidelines for Allowable Expenses 2, 3

Category Guidelines

Project Management Team Stipends

Not to exceed 10% of the project director’s salary and 5% of project leaders’ salaries. Teachers serving on the management team may be paid an honorarium at the same rate allowable for teacher stipends.

Contract Services

Not to exceed $50/presentation hour and $25/planning and preparation time for consultants or presenters ($800/day maximum); not to exceed $35/presentation hour and $17.50/planning and preparation time for district personnel ($560/day maximum). Only 2 hours prep time per hour of presentation time funded. (Time and Effort Logs Required.)

Higher Education Faculty Regular salary per hour of contact time; 50% of salary per hour of planning/preparation time. Only 2 hours prep time per hour of presentation time funded.

Project Management PD Reimburse travel expenses for management team participation in U.S. Dept. of ED and OSDE-hosted MSP events according to state/district guidelines.

External Evaluation Services

Not to exceed 8% of total project budget may be spent on a formal project external evaluator. OSDE will allow additional funds for a plan that successfully conducts a quasi- or experimental study following U.S. Dept. of ED guidelines/requirements. 4

Teacher’s Compensation Not to exceed $25 per hour during off-contract time; teacher fringe benefits may be covered by MSP grant funds. Teachers must be eligible to work in the United States.

Tuition

Annual tuition payment (payable to the IHE where the credits will be earned and coded) for graduate course credits is permissible if the course and participant meet all four of the following criteria: (1) the course is directly related to the MSP participants’ professional development plan; (2) the course will lead to the completion of an accredited graduate education program/endorsement; (3) the participant successfully completes the course with a grade of B or better; and (4) the tuition for a course is not already provided by the LEA.

Travel Reimburse mileage, meals, and lodging according to state/district guidelines for project-related travel.

Materials and Supplies Funds may be spent on materials and supplies to facilitate professional learning of teachers, not on classroom instructional materials for students of teacher participants.

Substitutes Determined by district substitute policy when MSP training sessions take place during teacher contract time.

Indirect Costs Not to exceed 8% of direct costs. (EDGAR §76.567(c)(2))2 All costs must be necessary, reasonable, and allocable. 3 MSP Program funds received must be used to supplement and not to supplant funds that would otherwise be used to support proposed activities.4 Quasi-experimental Study - A rubric designed by the U.S. Department of Education is used to determine whether a grantee’s evaluation meets the minimum criteria that need to be met for an evaluation to be successfully conducted and yield valid data. Evaluation components covered in the rubric include sample size, quality of measurement instruments, quality of data collection methods, data reduction rates, relevant statistics reported, and baseline equivalence of groups. The rubric is included in Appendix B of this document and is also posted at www.ed-msp.net under “Resources.”

16

Anticipated Grant Competition Timeline:The OSDE expects to adhere to the following timeline with respect to the MSP grant competition but reserves the right to make changes as necessary.

September 9, 2014 Request for Proposals (RFP) Published

September 9, 2014 Announcement of RFP Webinar

September 22-23, 2014(10:00 am - 11:00 am)

Technical Assistance Webinar Sessions:● MSP Technical Assistance Part 1 – September 22, 2014,

10:00 am – 11:00 am○ Understanding new elements of OK MSP, Partnerships,

and Needs Assessment https://oksdetraining.webex.com● Be sure to click on the “Upcoming” tab to select the

appropriate session

● MSP Technical Assistance Part 2 – September 23, 2014, 10:00 am – 11:00 am○ Work Plan, Evaluation, Budget

● https://oksdetraining.webex.com● Be sure to click on the “Upcoming” tab to select the

appropriate session These sessions will be taped and archived. Links to taped sessions

can be obtained by visiting the OSDE website STEM page: http://ok.gov/sde/stem

September 26, 2014 Notice of Intent to Apply Due

December 5, 2014 by 5:00 p.m

Proposals Received by the OSDE

December 2014 Proposal Review Panel

January 2015 Funding Recommendations to the SBE

January 2015 Announcement of Grant Awards

March 10, 2015 Required Spring Meeting for Project Directors - Location TBD

17

Section 3: Oklahoma MSP Program Key FeaturesProjects are expected to accomplish goals through several key features, which must be evident in all proposals: clearly defined partnerships, carefully delineated work plans, and comprehensive evaluation plans that employ both formative and summative measures. Key Features of the Oklahoma MSP ProgramPartnership Qualities:

The success of individual MSP projects rests squarely on the strength of the partner relationship. Each member of the project management team is expected to be actively engaged in the project effort at the institutional and individual levels, as well as share goals, responsibilities, and accountability for the program. The project management team must be convened regularly to oversee the design, implementation, and evaluation of the project. Evidence of Project Management Team meetings will be required. Furthermore, each partnership is expected to draw upon the expertise of all of its members through staff members’ collaborative facilitation of each MSP professional learning session. In addition to the expectations described above, partnerships should provide clear evidence of the following characteristics:

● Commitment: Partnership members should demonstrate commitment to project goals and projected outcomes unique to its proposal. Commitment is illustrated by each partner’s clear description of the expertise, time, and resources it will provide to support the goals of the partnership. Commitment is also evidenced by the descriptions of anticipated benefits included in each partner’s submitted Letter of Support. While matching funds are not required, in-kind support is highly desirable and preference will be given to proposals in which partners contribute their own resources, including the coordination of other applicable grants, toward the project’s success. Commitment is also evidenced in project management team meeting minutes and attendance records.

● Sustainability: Partnerships must provide a clear description of long-term plans to use project data to determine its impact on teaching and learning and to support the continuation of the project model beyond the duration of the grant.

● Capacity: Proposals must describe specific and achievable plans to recruit, serve, and retain a teacher cohort group with increased ability to improve student achievement in tested mathematics and science content areas. Further, proposals must provide a detailed description of the people and institutional resources available to conduct the project’s activities and how the expertise of each will contribute to the achievement of the project’s goals.

Project Work Plan Elements:

MSP project partnerships are expected to immerse teachers in a program of rigorous and appropriate courses and experiences that provide coherent study within a particular mathematics and/or science content area. Such programming should incorporate a number of elements:

● Scientifically-based Research: Project design must be informed by current research and studies on teaching and learning. Scientifically-based research involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs. This research base should provide a rationale for the chosen professional learning model.

● Cohort Approach: Projects must be designed to provide sustained professional learning opportunities to a cohort of teachers.

● Grade Bands: Projects may focus their efforts on mathematics and/or science teachers of grades K-5, 6-8, and/or 9-12 based on identified needs. Vertical teams or blends supported by needs and content are also appropriate. A separate needs assessment, work plan, and evaluation plan must be evident within the proposal for each grade band of teachers with whom the partnership proposes to work. If a blend of transitional grades

18

(for example 5-6 or 8-12 receiving the same instruction) is part of your plan, include a single needs assessment, work plan, and evaluation plan. Be very clear about why the grades will be grouped for common instruction.

● Professional Learning Plan Design: MSP projects must be designed to deliver at least 80 hours of ongoing professional learning to each teacher in the form of both intensive professional learning activities and follow-up training and classroom support. Intensive training is intended to improve the content knowledge and teaching skills of teachers while classroom follow-up training and support is intended to infuse the knowledge and skills gained directly into the classroom to benefit students. Classroom follow-up support and training must be directly related to the focus of the intensive training. Members from each of the partnership organizations must actively participate in both the classroom-level follow-up support as well as the intensive phase of the program. Of the 80 total hours (minimum) of training provided to each teacher per year, at least 60 must be devoted to intensive training and 20 to follow-up training and support.

Project Evaluation and Accountability Plan

Oklahoma’s MSP projects are expected to use both formative and summative assessment methods to evaluate effectiveness. In the formative sense, evaluation should provide evidence of the strengths and weaknesses of the program, informing the partnership’s understanding of what works and what does not in order to guide program modifications as needed. Such assessment should largely be provided by each project’s formal evaluator. In the summative sense, each project should determine which assessment tools will be utilized in evaluating and providing feedback on the overall success of the project as well as to inform individual partnerships of the effectiveness of the totality of their work. A list of resources is provided in Appendix A. Resources such as the National Science Foundation’s Math and Science Partnership Network (http://hub.mspnet.org) provides users access to program evaluations, assessment resources, and more. Applicants are encouraged to build a high-quality randomized controlled trial (RCT) into the design of their project in order to rigorously evaluate its effectiveness. RCTs are considered the gold standard for measuring a project’s impact based on persuasive evidence that (i) they are superior to other evaluation methods in producing valid estimates of a project’s impact; and (ii) the most commonly-used nonrandomized methods often produce erroneous conclusions. Applicants are encouraged to meet all Government Performance and Results Act (GPRA) criteria as defined in the Guide for Summarizing MSP Evaluation Designs and Results (Appendix B). This type of design must be carefully planned with an evaluator prior to submitting a proposal. MSP applicants, who by themselves may not have the required minimum sample of teachers to carry out an RCT, can propose to partner with other LEAs to form a consortium.

Providing Services to Eligible Nonpublic School Students, Teachers, and other PersonnelThe No Child Left Behind (NCLB) legislation, Section 9501, requires all applicants for certain discretionary grant programs to include and provide services to eligible nonpublic school students and/or teachers. Title 2 Part A is subject to the requirements of Sections 9501-9504 of the No Child Left Behind Act of 2001 regarding the equitable participation of nonpublic school teachers in this grant program.

● Nonpublic School Eligibility: Nonpublic school eligibility is based on the location of the nonpublic school(s), the design of the specific grant program and the needs of the nonpublic school(s). The needs must be able to be met via the discretionary grant program’s specific program design.* Generally, the nonpublic school must be located within the communities or geographic boundaries of the applicant agency or partner agency if applicable. According to the parameters of the grant program and available funding, the applicant agency determines the area to be served.

19

*Example: If the design of the grant program is to provide math instruction for seventh and eighth grade teachers, then the nonpublic school(s) must serve seventh and eighth grade teachers who are in need of math instruction and must be in the geographic area served by participating public schools.

● Timely and Meaningful Consultation: ○ The applicant agency is responsible to identify all appropriate nonpublic schools

and to contact the appropriate nonpublic school officials to begin the consultation process. The nonpublic school(s) must be given a genuine opportunity to participate in the grant program. The NCLB legislation requires all applicants to conduct timely and meaningful consultation with the appropriate nonpublic school officials prior to the development of the local project’s grant application and prior to any decision being made regarding the design of the local project that could affect the ability of nonpublic school students, teachers and other education personnel to receive benefits. Consultation must continue throughout the implementation and assessment of activities.

● Considerations: Listed below are the considerations that must be taken into account by all applicants when assessing the needs of the nonpublic school students and teachers and when determining in consultation with the nonpublic school(s) whether those needs fit the grant’s program design. Consultation generally must include discussion on such issues as:

○ what services will be provided;○ how, when, where, and by whom the services will be provided;○ how the services will be assessed and how the results of the assessment will be

used to improve those services;○ the amount of funds available for services; and○ how and when decisions about the delivery of services will be made.

● NOTE: A unilateral offer of services by an applicant agency with no opportunity for discussion on the part of the nonpublic school representative is not adequate consultation.

● Consistent and Comparable Services and Benefits: The NCLB legislation requires that the participation and involvement of the nonpublic school partners and participants be consistent (closely parallel, be similar) with the number of eligible children enrolled in nonpublic elementary and secondary schools within the geographic boundaries of the applicant agency or partner agency if applicable. The grant-related services and benefits must be comparable (having a similar effect) to those provided to public school children and teachers participating in the program, and they must be provided in a timely manner. All services to nonpublic school students and teachers must be secular, neutral, and non-ideological.

The Education Department General Administrative Regulations (EDGAR) §76.652 states that the applicant agency shall give appropriate representatives a genuine opportunity to express their views regarding each matter subject to the consultation requirements outlined above. By following this course of action, a successful consultation will result in a well-matched agreement between the applicant and the eligible nonpublic school(s). This agreement should:

○ be appropriate for the specific grant program;○ allow for the orderly and efficient integration of the services for the nonpublic

school students/teachers into the operation of the local project; and○ result in benefits which have similar effects for the applicant and the nonpublic

school students and/or teachers.● Use of Funds Requirements (EDGAR §76.650 - §76.662): When providing benefits to

nonpublic school students with federal funds, the following must be addressed:○ The grantee must maintain administrative control over all funds and property.

(No funds can flow directly to the nonpublic school via a subgrant).

20

○ The grantee may place equipment and supplies in the nonpublic school for the period of time needed for the grant. The grantee must ensure that the materials are used only for the purposes of the grant and can be removed from the nonpublic school without remodeling the nonpublic school facility.

○ Funds cannot be used for construction of nonpublic school facilities.○ Funds must be used to meet specific needs of students and staff. (Funds cannot

supplant benefits normally provided by the nonpublic school).○ Funds may be used to pay for services of an employee of the nonpublic school if

the employee performs the services outside of his or her regular hours and the employees performs the services under the supervision of the grantee.

○ All benefits provided, including equipment and materials, must be secular, neutral and non-ideological. (IASA, Sec 14503)

● Required Forms: The applicant must provide, as part of the application, the signed Equitable Participation of Nonpublic Schools found in the OSDE MSP Proposal Framework 2014-2015 posted at http://sde.ok.gov/stem.

An applicant agency may be disqualified from funding if it fails to provide this form.

21

FERPAThe Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. § 1232g; 34 CFR Part 99) is a Federal law that protects the privacy of student education records. The law applies to all schools that receive funds under an applicable program of the U.S. Department of Education. Generally, schools must have written permission from the parent or eligible student in order to release any information from a student's education record. However, FERPA allows schools to disclose those records, without consent, to the following parties or under the following conditions (34 CFR § 99.31):

● school officials with legitimate educational interest;● other schools to which a student is transferring;● specified officials for audit or evaluation purposes;● appropriate parties in connection with financial aid to a student;● organizations conducting certain studies for or on behalf of the school;● accrediting organizations;● to comply with a judicial order or lawfully issued subpoena;● appropriate officials in cases of health and safety emergencies; and● State and local authorities, within a juvenile justice system, pursuant to specific State law.

MSP Research is typically done in an established educational setting, involving normal educational practices, such as research on the effectiveness of instructional techniques [Section 97.101(b)(1)] involving the use of educational tests. Information is recorded in such a manner that human subjects cannot be identified [Section 97.101(b)(2)] and therefore does not require IRB (institutional review board) approval or parental notification. Please refer to the following website for additional information: http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html.

Awarded projects will receive additional training in updated FERPA rules at the required training session.

22

Section 4: Oklahoma MSP Proposal Format and SubmissionProposals submitted in response to this RFP must be prepared using the framework provided by the OSDE and submitted in accordance with the following guidelines.

Access the Proposal Framework here: http://sde.ok.gov/stem Format Requirements: Proposals that do not comply with these formatting requirements will not be reviewed or considered for funding.

● Typewritten and saved as a Microsoft Word read-only document or pdf file on a CD-ROM or jump drive

● Charts and graphs may be single spaced● Footer on each page with the page number, lead partner name, and appropriate grade band● Stapled or clipped in the upper left hand corner; no binders or folders● Include a cover page, which can be found in the framework document, and a table of contents● Required forms that are to be included in the body of the proposal are not subject to page

limitations. Page limitations apply to narrative sections only. Proposal Delivery: Partnerships must send one original of the complete proposal along with a CD-ROM or jump drive containing one complete proposal file in Microsoft Word (read-only) or PDF format.

● All proposals must be physically received by the OSDE by 5:00 p.m. on Friday, December 5, 2014. Incomplete, late, or incorrectly formatted proposals will not be scored or considered for funding.

● Applicants are encouraged to use express, certified, or registered mail.● Faxed or emailed proposals will not be accepted.● Mail proposals to:

Jeff Downs, M. Ed.

Executive Director of STEM2500 N. Lincoln Blvd., Suite 315

OKC, OK 73105

23

Section 5: Oklahoma MSP Proposal PreparationThe OSDE has prepared a comprehensive proposal framework to be used by all partnerships in preparing a proposal for funding consideration. Evident in this framework is the requirement for every partnership to include a separate needs assessment, work plan, and evaluation plan for each grade band of teachers with whom it intends to work. Furthermore, all proposals must include the following components, presented in the sequence specified below.

1. Grant Application Cover Page 2. Partner List3. Statement of Assurances (Including the Equitable Participation form)4. Application Preparation Checklist 5. Repeat Applicant Project Abstract

a. Partnerships that have previously received MSP Program funding must include an abstract of prior work. The abstract must describe the project’s intended goals, the amount of funding received, the number of teachers it intended to serve (according to its formal proposal), the number of teachers it actually served, an explanation of how the budget was spent, qualitative and quantitative evidence of progress towards goals, a description of partnership roles, and an indication of how the proposed work differs from, builds on, or is otherwise informed by prior efforts. The abstract may not exceed 3 single-spaced pages.

6. Project Abstracta. All partnerships must provide a 1-page, single-spaced abstract of the proposal that

briefly and concisely describes the program to be implemented and summarizes the intended results of the program. It should identify the project partners, the grade band(s) and content area(s) of proposed work, the number of teachers it intends to serve, the academic/instructional need of the schools in which they provide instruction, the partnership goals, and a brief overview of the work plan and evaluation plan.

7. Results of Needs Assessmenta. This section will identify and prioritize baseline professional learning needs of teachers in

partner school districts, disaggregated by grade level and content area. It must identify specific gaps or weaknesses in teacher content knowledge. This baseline information must be determined using a current (within the past 12 months) quantitative and qualitative content-driven assessment of teacher professional learning needs. It should also include a description of the methods used to collect this information. Additionally, the needs assessment must include the current status of student achievement in mathematics and/or science for the targeted grades and should be disaggregated in table form by gender, ethnicity, socioeconomic factors, English language learners (ELL), and disability and then analyzed in narrative form. It must clearly demonstrate high-need qualification.

b. The results of the teacher and student needs assessments must be used in the establishment of the goals and objectives for the proposed project.

c. Resource links to support the development of a Needs Assessment tool are provided in Appendix A.

8. Work Plan: Goals and Objectives, Project Action Plan, and Project Management Plana. Goals and Objectives – The project narrative must use the results of the content-driven

needs assessment to identify measurable project objectives for increasing teacher content

24

knowledge and changing teacher practice. It should describe the recruitment and retention strategies that will be used with the teacher cohort group. Objectives should be written at least as frequently as quarterly increments so projects may assess progress towards goals qualitatively and quantitatively. Objectives should be specific, measurable, attainable, realistic and time-bound (S.M.A.R.T.).

b. Project Action Plan – The project action plan should describe the proposed creative, strategic activities that extend beyond commonplace approaches and how they provide instruction to teachers at a level beyond the level of content they are expected to teach to students; model content-specific instructional strategies that will provide teachers with the methodologies to effectively improve student achievement; and describe how the professional learning sessions are specifically aligned to the content and curriculum in which participating teachers must provide instruction. The narrative should provide evidence of (a) an effective partnership among all organizations that work together to realize the project’s vision and goals, (b) the participation of the entire management team in planning, design, and implementation, and (c) sufficient capacity of the partners to support the scale and scope of the project, especially the number of teacher participants. It should describe in detail how the partnership will achieve the objectives and anticipated quantitative outcomes by means of a coherent plan. This description should include the research or evidence base on which the proposed work rests. It should describe how many schools and teachers will participate in the project and the level of need at those schools. Furthermore, it should describe how each partner will contribute to the proposed work. It must provide a timeline that correlates with the proposed action plan and the quantitative outcome goals and benchmarks.

c. Project Management Plan – This portion of the narrative should describe the management plan by which all partners are fully engaged to realize the partnership’s outcomes. It should describe in detail the specific roles, responsibilities, and time commitments of the project management team. It should also provide the number of STEM faculty and/or teacher preparation faculty who will be engaged in the project work. A one-page vitae for all project management team members, faculty members, and consultants involved with the project must be included in the proposal appendix. Their role in the grant should be noted on their vitae in the upper-right-hand corner.

9. Evaluation and Accountability Plana. The evaluation and accountability plan should be described in terms of how it will guide

project progress and will measure the impact of the work described in the action plan, including a description of the instruments/metrics by which the project will measure its progress towards objectives. It should describe both formative and summative assessment methods that will be used. Formative evaluation should provide evidence of the strengths and weaknesses of the project and help the partnership identify the extent to which the lessons learned from the sessions are being applied by teacher participants at the classroom level. Summative evaluation should give an objective analysis of qualitative and quantitative data, thus demonstrating the effectiveness of the project on student and teacher outcomes. If working with teachers in grades or courses that are not assessed by the State, you must state how you will monitor student growth. Although the evaluation plan will be developed with input from the partnership, objective analyses and findings must be determined by an external evaluator who is clearly separate and distinct from the partnership participants and their respective departments. A timeline for the

25

evaluation should be included, and the qualifications of the evaluator should be provided in a one-page vitae in the appendix.

i. Note: The needs assessment(s), work plan(s), and evaluation/accountability plan(s) for all grade bands included in the proposal must not exceed 20 total pages.

10. Budget and Budget Narrativea. Partnerships must submit one budget form for the entire proposed project. The amounts

requested for each budget line item should be documented and justified in the budget justification as specified below (section 10.b). The budget justification should be no more than three pages. Amounts and expenses budgeted also must be consistent with the proposing district’s policies and procedures and cost accounting practices used in accumulating and reporting costs. Cost principles governing the allowability of costs are contained in OMB Circulars A-21 (Colleges & Universities), A-87 (State, Local, & Indian Tribal Governments), and A-122 (Non-Profit Organizations) and are available at http://www.whitehouse.gov/omb/circulars/index.html.

b. The budget(s) and the corresponding narrative(s) should be aligned with the activities described in the proposal narrative, show evidence of effective, appropriate, and efficient use of funds, and describe clearly the full range of resources that will be used to accomplish the goals and objectives of the project.

11. Appendixa. Within the appendix of the proposal, partnerships should provide additional project

information including but not limited to i. partner identification forms, ii. affirmation of partnerships consultation,iii. bibliography of works cited, iv. 1-page vitae of appropriate partnership personnel, v. Memorandum of Understanding (MOU) from each partner, vi. letter of commitment and support from the lead applicant’s authorized

representative, andvii. additional proposal support information submitted at the discretion of the

partnership, such as samples of instruments used to conduct needs assessments, etc.

Note: Each MOU should clearly outline the role and contributions of the partner and provide evidence that the proposed partnership activities are integral to the partner’s instructional mission. It should be signed by the authorized authority (dean, VP, etc.) of each department of a higher education partner, the Superintendent of each partner school district, or the head of any other partner organization. All MOUs from school districts should clearly indicate their willingness to share aggregate student data of participating teachers in a timely fashion for annual reports to the US Department of Education.

26

Section 6: Oklahoma MSP Program Review and Award ProcessReview Process

The OSDE MSP Team will review proposals as they are received for eligibility, completeness, and compliance with application requirements. If, in the judgment of the OSDE MSP Team, a proposal is late or significantly incomplete, or if an applicant cannot establish its eligibility, the proposal will be omitted from consideration. In such cases, applicants will be notified of the decision in writing, and the decision of the OSDE MSP Team is final. An external review panel whose members have substantive expertise in mathematics and science will then be convened to review all eligible proposals. The OSDE MSP Team will recruit in-state and out-of-state panelists who bear no conflict of interest towards any of the partnerships. The review panel will use one of two scoring rubrics to evaluate the merits of each eligible proposal, assign a score, and make recommendations to the OSDE MSP Team in terms of program, budget, and efficacy. The review panel’s scores and recommendations will be the primary determinant of successful proposals and will form the basis for negotiation and final selection. Proposals will be ranked according to the final score assigned by the review panel and selected for funding consideration based upon specific criteria: final score; cost-effectiveness ratio determined by the relationship between the number of teachers served, the actual amount of teacher-faculty instructional contact time, and the total cost of the program; and geographic distribution. Following the external review, the OSDE MSP Team will submit award recommendations to the Oklahoma State Board of Education for the funding of those proposals that show the most promise for improving teacher content knowledge and instructional practice in mathematics and science. In order to maximize the effects of limited funds, applicants whose grants are awarded at less than the original request level may be asked to revise the project budget and/or scope of project work.

Review Criteria

The detailed scoring rubrics that will be used by the review panel to assess applicant proposals can be found in Appendices C and D of this RFP; however, the general review criteria are included on the following page. Any proposal that earns a score of zero in any of the efficacy of plan criteria on the scoring rubric(s) will be disqualified from funding consideration.

Additionally, the grade level notations in the points awarded column of both rubrics are included to make evident that scores from multiple grade band proposals will be averaged together to determine total criteria scores. Grade-bands not addressed by the project will not be included in the average score. If a project is working with transitional grades (i.e. 8th grade physical science and high school physical science teachers), then only one work plan is necessary. Do not create a middle grades plan and a high school plan if they are the same plan (same instructors, content, schedule, and cohort). Simply submit them in the category of the highest grade level and be clear about why common instruction for crossing grade bands is beneficial.

OSDE Priority Scoring PointsNine bonus points are available within the Priority Scoring Points section of each rubric. The

• Schools and teachers with greatest need and alignment with other strategic initiatives;• Meaningful administrator participation,• Creative, innovative approach.

27

Rubric #1: Criteria for New Applicants (not receiving previous funding):

Categories Points PossibleCommitment and Capacity of Partnership 9Demonstration of Need and Research Base 12Alignment of Goals/Objectives with Professional Learning Needs 18Efficacy of Plan 32Evaluation and Accountability Plan 20Budget and Cost Effectiveness 9OSDE Priority Scoring Points 9

Rubric #2: Criteria for Repeat Applicants (those receiving previous funding):

Categories Points PossibleEvidence of Prior MSP Project Work 11Commitment and Capacity of Partnership 9Demonstration of Need and Research Base 12Alignment of Goals/Objectives with Professional Learning Needs 15Efficacy of Plan 28Evaluation and Accountability Plan 16Budget and Cost Effectiveness 9OSDE Priority Scoring Points 9

Notification of Award

Upon completion of the review process, the OSDE MSP Team will present funding recommendations to the State Board of Education (SBE) for its consideration. Once final funding decisions have been approved by the SBE, project directors will be notified of the status of their proposal in writing. A required meeting of all project directors and leadership teams of funded partnerships will be held by the OSDE MSP Team on March 10, 2015. If you submit a proposal to OSDE, please save this date in case your grant is awarded.

Please direct all MSP Program inquiries to: Jeff Downs

405-521-2107Jeff.Downs@sde.ok.gov

28

Appendix A: Possible Resources for MSP Proposal Preparation

Resource: Link MSP Planning Evaluation Math ScienceU.S. Department of Education/MSP Program - http://www.ed.gov/programs/mathsci/index.html

X

U.S. Department of Education’s Mathematics and Science Partnerships (MSP) - http://www.ed-msp.net

X X X

National Science Foundation’s MSP Network - http://hub.mspnet.org/index.cfm/home X X X X X

A Guide for Reporting on Rigorous Evaluations for the US Department of Education Mathematics and Science Partnerships (MSP)http://www.ed-msp.net/public_documents/document/resource/Guide%20for%20Reporting%20on%20MSP%20Evaluations.pdf

X X

The Oklahoma Department of Educationhttp://sde.ok.gov and http://www.ok.gov/sde/oklahoma-academic-standards

X X X

The Oklahoma Science Teachers Associationhttp://www.oklahomascienceteachersassociation.org/

The Oklahoma Council of Teachers of Mathematicshttp://okctm.org

American Association for the Advancement of Science Project 2061 Science Assessment - http://assessment.aaas.org

X X X

Learning Mathematics for Teaching (LMT) - http://sitemaker.umich.edu/lmt/links

X X X

The Mathematics Assessment Program (MAP) - http://map.mathshell.org/materials/index.php

X X X

Horizon Research, Incorporated (HRI) - http://www.horizon-research.com/

X X X X

Learning Forward (Formerly the National Staff Development Council) - http://learningforward.org

X

Project MOSART - https://www.cfa.harvard.edu/smgphp/mosart

X X

National Council of Teachers of Mathematics (NCTM) - http://www.nctm.org

X X

National Science Teachers Association (NSTA) - http://www.nsta.org

X X

National Academies and Board on Science Education - http://sites.nationalacademies.org/DBASSE/BOSE/Framework_K12_Science/index.htm

X X

29

Appendix B: Guide for Summarizing MSP Evaluation Designs and ResultsOne of the goals of the Mathematics and Science Partnership (MSP) program is to contribute to the knowledge base on effective professional development in mathematics and science. To this end, the MSP legislation (Title II, Part B of the No Child Left Behind Act) requires every MSP project to design and implement an evaluation and accountability plan that allows for a rigorous assessment of its effectiveness, and which includes information on the project’s impact on teachers and students. In order to ensure that projects are providing high-quality information on program outcomes, the Criteria for Classifying Designs of MSP Evaluations (printed after this document /part of Appendix B), was developed as part of the Data Quality Initiative through the Institute for Education Sciences (IES) at the U.S. Department of Education. The criteria that comprise the rubric specify conditions that projects that use experimental designs and quasi-experimental designs must meet in order to be deemed rigorous evaluations. In 2008, the rubric was applied to the final evaluation reports of completed MSP projects for the first time. In doing so, it became apparent that most projects evaluate more than one component of their project (e.g., teacher content knowledge in mathematics and/or science, teacher attitudes and beliefs, student content knowledge in mathematics and/or science), that different evaluation techniques are often applied to the different components, and that some components meet all the criteria for being classified as a rigorous evaluation while other components do not. It also became apparent that while most projects collect most of the information needed to assess their evaluation design(s), few report the information in a manner that allows it to be easily evaluated with the rubric. This Guide was developed to provide Project Directors and Evaluators with guidance on how best to summarize their evaluation data to facilitate the review and assessment of their evaluation design(s). We recommend that you present the results for each of the criteria discussed below in an Executive Summary at the beginning of your final evaluation report. Screening ProcessMSP evaluations undergo a two-stage screening process. They are first screened for the type of evaluation design and then for the strength of the implementation of the individual elements of the design. Below we present the criteria that are used in each stage of the screening process followed by recommended summary tables or narrative reporting guidelines, where relevant, for presenting information about your evaluation. Evaluation DesignTo be classified as having a strong design, only one component of the evaluation has to be either 1) an experimental study that compares the outcomes of a randomly assigned treatment and control group or 2) a quasi-experimental study that compares the outcomes of a treatment and comparison group that meets one of two design criteria:

1. comparison group study with equating (matching) — statistical controls or matching techniques were used to make the treatment and comparison groups similar on their pre-intervention characteristics; or

2. regression-discontinuity study — individuals (or other units such as classrooms or schools) were assigned to treatment or comparison groups on the basis of a “cutoff” score on a pre-intervention non-dichotomous measure.

30

Summary InformationList each outcome that your are evaluating and the participant group to whom it applies, and check the type of evaluation design applied to that group. The table below provides an example of a project that evaluated five outcomes using three different designs.

Table 1: Evaluation Design TypeTable 1: Evaluation Design TypeTable 1: Evaluation Design TypeTable 1: Evaluation Design TypeTable 1: Evaluation Design Type

Participant Group and OutcomeExperimental

Design

Quasi-Experimental DesignQuasi-Experimental DesignOther

Design Participant Group and Outcome

Experimental

Design with equating(matching)

regression discontinuity

Other

Design

Elementary teachers science knowledge X

Elementary teachers mathematics

knowledgeX

Elementary students science

achievement X

Elementary students mathematics

achievement X

Elementary teacher classroom practice

science X

Elementary teacher classroom practice

mathematics X

Experimental DesignsFor each participant group and outcome that was evaluated using an experimental design, please describe how units (i.e., participants, classroom schools, or districts) were randomly assigned to groups.

1. Participant Group/Outcome: ____________________: (describe random assignment) 2. Participant Group/Outcome: ____________________: (describe random assignment) 3. Participant Group/Outcome: ____________________: (describe random assignment) Elements of the DesignTo be classified a strong design each participant group/outcome that was evaluated using a quasi-experimental design must meet all of the following six criteria. Participant group/outcomes that were evaluated using an experimental design must meet every criterion except the first, baseline equivalence of groups, as randomly assigned groups are assumed to be equivalent at baseline.

31

A. Baseline Equivalence of Groups (quasi-experimental designs only)

Criterion:● No significant pre-intervention differences between treatment and comparison group on variables

related to the study’s key outcomes; or● Adequate steps were taken to address the lack of baseline equivalence in the statistical analysis.

Summary Information:For each participant group and outcome provide the treatment and comparison groups’ pre-test score (mean or percent) and the p-value of the statistical test used to assess equivalence.

Table 2: Baseline Equivalence of GroupsTable 2: Baseline Equivalence of GroupsTable 2: Baseline Equivalence of GroupsTable 2: Baseline Equivalence of Groups

Participant Group/Outcome and Matching Variables

Treatment GroupPre-test Score

Comparison Group Pre-test

Score p-value

Participant Group and Outcome: Middle School Students/Middle School MathematicsParticipant Group and Outcome: Middle School Students/Middle School MathematicsParticipant Group and Outcome: Middle School Students/Middle School MathematicsParticipant Group and Outcome: Middle School Students/Middle School Mathematics

Student achievement mean or percent mean or percent

Student demographic characteristics mean or percent mean or percent

Participant Group and Outcome: Middle Schools Students/Middle School ScienceParticipant Group and Outcome: Middle Schools Students/Middle School ScienceParticipant Group and Outcome: Middle Schools Students/Middle School ScienceParticipant Group and Outcome: Middle Schools Students/Middle School Science

Student achievement mean or percent mean or percent

Student demographic characteristics mean or percent mean or percent

Participant Group and Outcome: Middle School Teachers/Middle School ScienceParticipant Group and Outcome: Middle School Teachers/Middle School ScienceParticipant Group and Outcome: Middle School Teachers/Middle School ScienceParticipant Group and Outcome: Middle School Teachers/Middle School Science

Teacher characteristics mean or percent mean or percent

Participant Group and Outcome: Middle School Teachers/Middle School ScienceParticipant Group and Outcome: Middle School Teachers/Middle School ScienceParticipant Group and Outcome: Middle School Teachers/Middle School ScienceParticipant Group and Outcome: Middle School Teachers/Middle School Science

Teacher characteristics mean or percent mean or percent

B. Sample Size

Criterion:● Sample size was adequate based on a power analysis with recommended:

○ Significance level = 0.05○ Power = 0.8○ Minimum detectable effect informed by actual data; or

Absent a power analysis, a study will qualify as meeting the criterion in the following scenarios assuming the level of the intervention is the same as the unit of assignment or grouping (see Working Definitions

32

for Projects in Criteria section at the end of this document (part of this Appendix B) for the assumptions that each scenario is based on). Teacher Outcomes

● Case #1: For interventions at the school or district level, an evaluation would need a sample of at least 12 schools or districts.

● Case #2: For interventions at the teacher or classroom level, an evaluation would need a sample of at least 60 teachers

Student outcomes● Case #1: For interventions at the school or district level, an evaluation would need a sample

of at least 12 schools or districts.● Case #2: For interventions at the teacher or classroom level, an evaluation would need a

sample of at least 18 classrooms/teachers.● Case #3: For interventions at the individual student level, an evaluation would need a

sample of at least 130 students. If the design is unbalanced (i.e., there are more treatment units than control/comparison or vice versa), the smaller of the two groups must at least meet the minimum sample size divided by 2. For example, for teacher outcomes Case #1, it is acceptable if there are 6 control/comparison schools and more than 6 treatment schools or vice versa.

Summary Information:For each participant group and outcome provide the final sample size at the level of random assignment or matching for the treatment and comparison/control group. Provide power calculation assumptions, if applicable.

Table 3: Sample SizeTable 3: Sample SizeTable 3: Sample SizeTable 3: Sample Size

Participant Group and Outcome

Treatment Group(Final sample size)

Comparison/Control Group

(Final sample size)

Power Analysis Findings

(if applicable)

Elementary teachers

mathematics knowledge N N

alpha =power =MDE =

Elementary students science

achievement N N

alpha =power =MDE =

Elementary students

mathematics achievement N N

alpha =power =MDE =

Elementary teacher

classroom practice science N N

alpha =power =MDE =

Recommended significant levels: alpha = 0.05, power = 0.8; minimal detectable effect (MDE) = informed by

actual data.

Recommended significant levels: alpha = 0.05, power = 0.8; minimal detectable effect (MDE) = informed by

actual data.

Recommended significant levels: alpha = 0.05, power = 0.8; minimal detectable effect (MDE) = informed by

actual data.

Recommended significant levels: alpha = 0.05, power = 0.8; minimal detectable effect (MDE) = informed by

actual data.

33

C. Quality of Measurement Instruments

Criterion:● The study used existing data collection instruments that had already been deemed valid and

reliable to measure key outcomes; or● The study used data collection instruments developed specifically for the study that were

sufficiently pre-tested with subjects who were comparable to the study sample.

Using selected items from a validated and reliable instrument or instruments is acceptable if the resulting instrument:

● Includes at least 10 items, and● At least 70 percent of the items are from the validated and reliable instrument(s).

Summary Information:For each participant group and outcome, provide the name of the instrument that was used to measure the outcome and provide evidence of the instrument’s validity and reliability. The evidence for borrowed or adapted instruments may be a website or other reference where the evidence is provided, or a narrative description of the evidence. For locally developed instruments that pre-tested the instruments, provide evidence of reliability and validity from those tests. For locally developed instruments that use items from one or more pre-existing valid and reliable instruments, provide the total number of items and the number of items borrowed from each instrument. The table below provides examples of how to present data on different types of instruments.

Table 4: Data Collection InstrumentsTable 4: Data Collection InstrumentsTable 4: Data Collection Instruments

Participant Group and Outcome Name of InstrumentEvidence for Validity and

Reliability

Teacher content knowledge – mathematics

DTAMSCite website or other reference where evidence can be found.

Teacher content knowledge – mathematics

Locally developed instrument

Total items = 20NAEP items = 15LMT items = 5

Teacher content knowledge – physicsLocally developed instrument

Narrative description of evidence (e.g., Cronbach alpha, face validity).

Teacher content knowledge - biologyLocally developed instrument

Not tested for validity or reliability.

34

D. Quality of the Data Collection Methods

Criterion:

● The methods, procedures, and timeframes used to collect the key outcome data from treatment and comparison groups were the same.

Summary Information:For each participant group and outcome, describe the method/procedure for collecting data from the treatment group, and indicate whether the same method/procedure was used to collect data from the comparison group. If the same method was not used, describe the method/procedure.

4. Participant Group and Outcome: ______________________a. Method/procedure for collecting data from treatment group (describe):

b. Was the same method/procedure used to collect data from the comparison group? ____ Yes ___ Noi. If no, please describe how the method/procedure was different:

c. Time frame for data collection. Indicate the month and year that each test was administered to each group.

Table 5: Time Frame for Data CollectionTable 5: Time Frame for Data CollectionTable 5: Time Frame for Data CollectionTable 5: Time Frame for Data Collection

Participant Group and

Outcome

Month and YearMonth and YearMonth and Year

Pre-test Post-test Repeated Post-test

Treatment group

Comparison group

2. Participant Group and Outcome: ______________________

a. Method/procedure for collecting data from treatment group (describe):

b. Was the same method/procedure used to collect data from the comparison group? ____ Yes ___ Noi. If no, please describe how the method/procedure was different:

c. Time frame for data collection. Indicate the month and year that each test was administered to each group.

Table 5: Time Frame for Data CollectionTable 5: Time Frame for Data CollectionTable 5: Time Frame for Data CollectionTable 5: Time Frame for Data Collection

Participant Group and

Outcome

Month and YearMonth and YearMonth and Year

Pre-test Post-test Repeated Post-test

Treatment group

Comparison group

35

3. Participant Group and Outcome: ______________________a. Method/procedure for collecting data from treatment group (describe):

b. Was the same method/procedure used to collect data from the comparison group? ____ Yes ___ No i. If no, please describe how the method/procedure was different:

c. Time frame for data collection. Indicate the month and year that each test was administered to each group.

Table 5: Time Frame for Data CollectionTable 5: Time Frame for Data CollectionTable 5: Time Frame for Data CollectionTable 5: Time Frame for Data Collection

Participant Group and

Outcome

Month and YearMonth and YearMonth and Year

Pre-test Post-test Repeated Post-test

Treatment group

Comparison group

E. Data Reduction Rates

There are two aspects to the data reduction criterion: attrition rates and response rates. An evaluation must meet the criterion for both attrition and response rates in order for it to meet the data reduction rates criterion. One exception is for cross-sectional studies that collect one-time data when only response rates apply. For longitudinal/pre-post studies that collect data from the same individuals over time, one needs to look at both the response rates and attrition rates criteria. Criterion:

● The study measured the key outcome variable(s) in the post-tests for at least 70 percent of the original study sample (treatment and comparison groups combined); OR

● there is evidence that the high rates of data reduction were unrelated to the intervention; AND● The proportion of the original study sample that was retained in the follow-up data collection

activities (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was similar for both the treatment and comparison groups (i.e., less than or equal to a 15 percent difference); OR

● the proportion of the original study sample that was retained in the follow-up data collection was different for the treatment and comparison groups, and sufficient steps were taken to address this differential attrition were not taken in the statistical analysis.

Summary Information:For each participant group and outcome, provide the following information for the treatment and comparison group: original sample size, pre-test sample size and the pre-test response rate (the percent of the pre-test sample that took the pre-test), post-test sample size and post-test response rate (the percent of the post-test sample that took the post-test), and the attrition rate, where the rate is calculated as the number of individuals who took both the pre- and post-test divided the number of individuals who took the post test.

36

Table 6: Data Reduction Rates Table 6: Data Reduction Rates Table 6: Data Reduction Rates Table 6: Data Reduction Rates Table 6: Data Reduction Rates

Original

Sample Size

Pre-test Sample Size

& Response Rate

Post-test Sample

Size & Response

Rate

Attrition Rate (for

designs with pre-

test)

Participant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers science

Treatment group N N, % responding N, % responding %

Comparison group N N, % responding N, % responding %

Participant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematics

Treatment group N N, % responding N, % responding %

Comparison group N N, %responding N,% responding %

Participant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students science

Treatment group N N, % responding N, % responding %

Comparison group N N, % responding N, % responding %

Participant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematics

Treatment group N N, % responding N, % responding %

Comparison group N N, % responding N, % responding %

E. Relevant Statistics

Criterion:● The final report includes treatment and comparison group post-test means and tests of

significance for key outcomes, OR● Pprovides sufficient information for calculation of statistical significance (e.g., mean, sample

size, standard deviation/standard error). Summary Information: For each participant group and outcome, provide the following information for the treatment and comparison group: post-test sample size, mean or percent, and test of significance; or post-test sample size, mean or percent, and standard deviation (SD) or standard error (SE).

Table 7: Relevant StatisticsTable 7: Relevant StatisticsTable 7: Relevant StatisticsTable 7: Relevant StatisticsTable 7: Relevant StatisticsTable 7: Relevant Statistics

Post-test N Mean or Percent SD or SE t,F, or chi square p-value

Participant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers scienceParticipant Group and Outcome: Elementary teachers science

Treatment group

Comparison group

Participant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematicsParticipant Group and Outcome: Elementary teachers mathematics

Treatment group

Comparison group

Participant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students scienceParticipant Group and Outcome: Elementary students science

37

Treatment group

Comparison group

Participant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematicsParticipant Group and Outcome: Elementary students mathematics

Treatment group

Comparison group

Criteria for Classifying Designs of MSP Evaluations2● Experimental study — the study measures the intervention’s effect by randomly assigning

individuals (or other units, such as classrooms or schools) to a group that participated in the intervention, or to a control group that did not; and then compares post-intervention outcomes for the two groups

● Quasi-experimental study — the study measures the intervention’s effect by comparing post-intervention outcomes for treatment participants with outcomes for a comparison group (that was not exposed to the intervention), chosen through methods other than random assignment. For example:

○ Comparison-group study with equating — a study in which statistical controls and/or matching techniques are used to make the treatment and comparison groups similar in their pre-intervention characteristics

○ Regression-discontinuity study — a study in which individuals (or other units, such as classrooms or schools) are assigned to treatment or comparison groups on the basis of a “cutoff” score on a pre-intervention non-dichotomous measure

● Other — the study uses a design other than a randomized controlled trial, comparison-group study with equating, or regression-discontinuity study, including pre-post studies, which measure the intervention’s effect based on the pre-test to post-test differences of a single group, and comparison-group studies without equating, or non-experimental studies that compare outcomes of groups that vary with respect to implementation fidelity or program dosage.

38

2 To be used for addressing following MSP GPRA measure: The percentage of MSP projects that use an experimental or quasi-experimental design for their evaluations that are conducted successfully and that yield scientifically valid results.

Criteria for Assessing whether Experimental DesignsWere Conducted Successfully and Yielded Scientifically Valid ResultsA. Sample size 3

• Met the criterion — sample size was adequate (i.e. based on power analysis with recommended significance level=0.05, power=0.8, and a minimum detectable effect informed by the literature or otherwise justified).

• Did not meet the criterion — the sample size was too small• Did not address the criterion

B. Quality of the Measurement Instruments● Met the criterion — the study used existing data collection instruments that had already been

deemed valid and reliable to measure key outcomes; or data collection instruments developed specifically for the study were sufficiently pre-tested with subjects who were comparable to the study sample

● Did not meet the criterion — the key data collection instruments used in the evaluation lacked evidence of validity and reliability

● Did not address the criterionC. Quality of the Data Collection Methods

● Met the criterion — the methods, procedures, and timeframes used to collect the key outcome data from treatment and control groups were the same

● Did not meet the criterion — instruments/assessments were administered differently in manner and/or at different times to treatment and control group participants

D. Data Reduction Rates (i.e. Attrition Rates, Response Rates)● Met the criterion — (1) the study measured the key outcome variable(s) in the post-tests for at

least 70 percent of the original study sample (treatment and control groups combined) or there is evidence that the high rates of data reduction were unrelated to the intervention, AND (2) the proportion of the original study sample that was retained in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was similar for both the treatment and control groups (i.e. less or equal to a 15-percent difference), or the proportion of the original study sample that was retained in the follow-up data collection was different for the treatment and control groups, but sufficient steps were taken to address this differential attrition in the statistical analysis.

● Did not meet the criterion — (1) the study failed to measure the key outcome variable(s) in the post-tests for 30 percent or more of the original study sample (treatment and control groups combined), and there is no evidence that the high rates of data reduction were unrelated to the intervention; OR (2) the proportion of study participants who participated in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was significantly different for the treatment and control groups (i.e. more than a 15-percent difference) and sufficient steps to address differential attrition were not taken in the statistical analysis

● Did not address the criterionE. Relevant Statistics Reported

39

3 The critical sample size here is related to the unit of assignment. For example, if the assignment is made at the school level, the relevant sample size is the number of schools involved.

● Met the criterion — the final report includes treatment and control group post-test means, and tests of statistical significance for key outcomes; or provides sufficient information for calculation of statistical significance (e.g., mean, sample size, standard deviation/standard error)

● Did not meet the criterion — the final report does not include treatment and control group post-test means, and/or tests of statistical significance for key outcomes; or provide sufficient information for calculation of statistical significance (e.g., mean, sample size, standard deviation/standard error)

● Did not address the criterion

40

Criteria for Assessing whether Quasi-Experimental DesignsWere Conducted Successfully and Yielded Scientifically Valid Results A. Baseline Equivalence of Groups

● Met the criterion — there were no significant pre-intervention differences between treatment and comparison group participants on variables related to the study’s key outcomes; or adequate steps were taken to address the lack of baseline equivalence in the statistical analysis

● Did not meet the criterion — there were statistically significant pre-intervention differences between treatment and comparison group participants on variables related to the study’s key outcomes; and no steps were taken to address lack of baseline equivalence in the statistical analysis

● Did not address the criterionB. Sample size 4

● Met the criterion — sample size was adequate (i.e. based on power analysis with recommended significance level=0.05, power=0.8, minimum detectable effect size informed by the literature or otherwise justified)

● Did not meet the criterion — the sample size was too small● Did not address the criterion

C. Quality of the Measurement Instruments● Met the criterion — the study used existing data collection instruments that had already been

deemed valid and reliable to measure key outcomes; or data collection instruments developed specifically for the study were sufficiently pre-tested with subjects who were comparable to the study sample

● Did not meet the criterion — the key data collection instruments used in the evaluation lacked evidence of validity and reliability

● Did not address the criterionD. Quality of the Data Collection Methods

● Met the criterion — the methods, procedures, and timeframes used to collect the key outcome data from treatment and comparison groups were the same.

● Did not meet the criterion — instruments/assessments were administered differently in manner and/or at different times to treatment and comparison group participants.

● Did not address the criterionE. Data Reduction Rates (i.e. Attrition Rates, Response Rates)

● Met the criterion — (1) the study measured the key outcome variable(s) in the post-tests for at least 70 percent of the original study sample (treatment and comparison groups combined) or there is evidence that the high rates of data reduction were unrelated to the intervention, AND (2) the proportion of the original study sample that was retained in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was similar for both the treatment and comparison groups (i.e. less or equal to a 15-percent difference), or the proportion of the original study sample that was retained in the follow-up data collection was different for the treatment and comparison groups, and sufficient steps were taken to address this differential attrition were not taken in the statistical analysis.

● Did not meet the criterion — (1) the study failed to measure the key outcome variable(s) in the post-tests for 30 percent or more of the original study sample (treatment and comparison groups

41

4 The critical sample size here is related to the unit of assignment. For example, if the assignment is made at the school level, the relevant sample size is the number of schools involved.

combined), and there is no evidence that the high rates of data reduction were unrelated to the intervention; OR (2) the proportion of study participants who participated in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was significantly different for the treatment and comparison groups (i.e. more than a 15-percent) and sufficient steps were not taken to address differential attrition in the statistical analysis.

● Did not address the criterion F. Relevant Statistics Reported

● Met the criterion — the final report includes treatment and comparison group post-test means, and tests of statistical significance for key outcomes; or provides sufficient information for calculation of statistical significance (e.g., mean, sample size, standard deviation/standard error).

● Did not meet the criterion — the final report did not include treatment and comparison group post-test means, or tests of statistical significance for key outcomes; or provide sufficient information for calculation of statistical significance (e.g., mean, sample size, standard deviation/standard error).

● Did not address the criterion

42

MSP Rubric Working Definitions for Projects The section contains working definitions to help interpret criteria in the Criteria for Classifying Designs for MSP Evaluations rubric. Eligibility of evaluation reportOnly final evaluation reports that contain post-test results on key outcomes will be evaluated. The review focuses exclusively on components regarding program impact, and does not cover assessment of implementation fidelity or performance against benchmarks. Definition of an evaluationAn evaluation design may contain multiple outcomes. For the purpose of implementing this rubric, the major outcomes of interest are 1) teacher content knowledge, 2) teacher instructional practices, and 3) student achievement. The reviewer will apply each rubric criterion as it relates to the three outcomes separately. Data collected on the three outcomes of interest might come from teachers/students in various grades and use different designs. If the implementation of the study design for an outcome meets all the criteria for at least one grade, the design for that outcome is considered as meeting the criteria. For example, if a study of 4th grade math achievement met the criteria but a study of 5th grade math did not, the student achievement evaluation from the project will be considered meeting the criteria based on the merit of its 4th grade math achievement study. Baseline equivalence of groupsVariables related to key outcomes may vary. For example, if the key student outcome is achievement, the most relevant variable will be an achievement outcome from the same or similar test conducted prior to the intervention. Other related variables, although not equally effective, can be related to student socio-economic status. If the key outcome is teacher effectiveness, the most relevant variables will be measures of teacher effectiveness from the same or similar pre-test. Other related variables may include measures of teacher quality such as level of education and/or years of teaching experience. Sample sizeThe sample size refers to the final sample size; that is the sample for which data have been collected.

Absent a power analysis, a study will qualify as “Met the criterion” in the following scenarios assuming the level of intervention is the same as the unit of assignment/grouping: Teacher outcomes

● Case #1: For interventions at the school or district level, an evaluation would need a sample of at least 12 schools/districts based on following assumptions: 1) a balanced sampling design that randomizes/matches at the school/district level; 2) 0.05 level of significance in a two-tailed test; 3) a minimum detectable effect size of 0.50; 4) the power of the test is 0.8; 5) each school/district has at least 15 teachers; 6) intraclass correlation of 0.05; and 7) a school/district level covariate (i.e. aggregated pre-test score) explains 70 percent of the variation.

● Case #2: For interventions at the teacher or classroom level, an evaluation would need a sample of at least 60 teachers based on following assumptions: 1) a balanced sampling design that randomizes/matches at the teacher/classroom level; 2) 0.05 level of significance in a two-tailed

43

test; 3) a minimum detectable effect size of 0.50; 4) the power of the test is 0.8; and 5) a teacher/classroom level covariate (i.e. pre-test score) explains 70 percent of the variation.

Student outcomes

● Case #1: For interventions at the school or district level, an evaluation would need a sample of at least 12 schools or districts based on following assumptions: 1) a balanced sampling design that randomizes/matches at school/district level; 2) 0.05 level of significance in a two-tailed test; 3) a minimum detectable effect size of 0.35; 4) the power of the test is 0.8; 5) each school or district has at least 75 students; 6) intraclass correlation of 0.05; and 7) a school/district level covariate (i.e. aggregated pre-test score) explains 70 percent of the variation.

● Case #2: For interventions at the teacher or classroom level, an evaluation would need a sample of at least 18 classrooms/teachers based on following assumptions: 1) a balanced sampling design that randomizes/matches at the classroom/teacher level; 2) 0.05 level of significance in a two-tailed test; 3) a minimum detectable effect size of 0.35; 4) the power of the test is 0.8; 5) each class has at least 25 students; 6) intraclass correlation of 0.05; and 7) a class/teacher level covariate (i.e. aggregated pre-test score) explains 70 percent of the variation.

● Case #3: For interventions at the individual student level, an evaluation would need a sample of at least 130 students based on following assumptions: 1) a balanced sampling design that randomizes/matches at the student level; 2) 0.05 level of significance in a two-tailed test; 3) a minimum detectable effect size of 0.35; 4) the power of the test is 0.8; and 5) a student level covariate (i.e. pre-test score) explains 70 percent of the variation.

If the design is unbalanced (i.e., there are more treatment units than control/comparison or vice versa), the smaller of the two groups must at least meet the minimum sample size divided by 2. For example, for teacher outcomes case #1, it is acceptable if there are 6 control/comparison schools and more than 6 treatment schools or vice versa. Quality of measurement instrumentsIf the evaluators used an existing state accountability assessment or other widely used assessments (i.e. Iowa test, TerraNova) in totality one can assume that their psychometric properties are adequate. Using selected items from a validated and reliable instrument or instruments is acceptable if the resulting instrument includes at least 10 items and at least 70 percent of the items are from the validated and reliable instrument(s). In addition, all instruments should at least have face validity. Data reduction ratesThere are two aspects to the data reduction criterion: attrition rates and response rates. An evaluation must meet the criterion for both attrition and response rates in order for it to meet the data reduction rates criterion. One exception is for cross-sectional studies that collect one-time data. For cross-sectional studies only response rates apply. For longitudinal/pre-post studies that collect data from the same subject over time, one needs to look at both the response rates and attrition rates criteria. If not provided in the report, the rates can be loosely calculated a) attrition rates b) response rates:

a. Posttest N/ Pretest Nb. Posttest N/ Original N

44

The first component of the criterion refers to overall data reduction and the second is related to differential reduction (i.e., between treatment and control/comparison groups). If the 70-percent data retention rate is not met, an evaluation may meet the criterion if the evaluators provide valid explanations (e.g., the schools are located in high mobility areas) or have addressed potential differences between sample members who have post-test data and those who do not in the analysis. ReferencesRaudenbush, S.W., Spybrook, J., Liu, X, and Cogndon, R. (2005). Optimal design for longitudinal and

multilevel research.

45

Appendix C: Scoring Rubric for Applicants Not Receiving Funds in the Previous Cycle

Scoring Rubric for MSP 2014-2015 New Applicant Proposals

Criterion A: Commitment and Capacity of Partnership (9 Possible Points)Criterion A: Commitment and Capacity of Partnership (9 Possible Points)Criterion A: Commitment and Capacity of Partnership (9 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Does the project management team have the expertise to implement and sustain a math and/or science professional learning program? Do individuals who planned the project represent the primary partners i.e. LEA and IHEs? Is there evidence that mathematicians, scientists, and/or engineers are playing major roles in the design and delivery of the proposed program? Are the roles of all partners clearly identified? Does the work plan engage all partners in meaningful ways? Is there evidence that the partners share goals, responsibilities, and accountability for the proposed work? Does the governance structure describe communication, decision-making, and fiscal responsibilities among the project partners?

Guiding Questions: Does the project management team have the expertise to implement and sustain a math and/or science professional learning program? Do individuals who planned the project represent the primary partners i.e. LEA and IHEs? Is there evidence that mathematicians, scientists, and/or engineers are playing major roles in the design and delivery of the proposed program? Are the roles of all partners clearly identified? Does the work plan engage all partners in meaningful ways? Is there evidence that the partners share goals, responsibilities, and accountability for the proposed work? Does the governance structure describe communication, decision-making, and fiscal responsibilities among the project partners?

Guiding Questions: Does the project management team have the expertise to implement and sustain a math and/or science professional learning program? Do individuals who planned the project represent the primary partners i.e. LEA and IHEs? Is there evidence that mathematicians, scientists, and/or engineers are playing major roles in the design and delivery of the proposed program? Are the roles of all partners clearly identified? Does the work plan engage all partners in meaningful ways? Is there evidence that the partners share goals, responsibilities, and accountability for the proposed work? Does the governance structure describe communication, decision-making, and fiscal responsibilities among the project partners?

________________________

9________________________

9________________________

9________________________

9

Exceeds Standard (3 Pts. each) Meets Standard (2 Pts. each) Below Standard (1 Pt. each) K-5 6-8 9-12 Avg.

Strong evidence of the number and quality of staff from the primary partners to carry out the proposed activitiesQualifications are provided for key partners’ staff and appear to be exceedingly strong

Adequate number and quality of staff from the primary partners to carry out the proposed activitiesQualifications of key partners’ staff are described and appear to be acceptable

Little evidence of the number and quality of staff from the primary partners to carry out the proposed activitiesQualifications of key partners’ staff are described but appear to be limited

Shows long term commitment of partnersInstitutional resources are given in detail

Shows commitment of partners;Institutional resources are given acceptably

Shows somewhat limited commitment of partnersInstitutional resources are given but without detail

Project is likely to impact a high percentage (>50%) of teachers in need

Project is likely to impact an acceptable percentage (25%-50%) of teachers in need

Project is likely to impact a limited percentage (<25%) of teachers

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion B: Demonstration of Need and Research Base (12 Possible Points)Criterion B: Demonstration of Need and Research Base (12 Possible Points)Criterion B: Demonstration of Need and Research Base (12 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Are planned activities supported by current research on effective professional learning practices and mathematics or science learning? Is that research cited in the proposal? Does the proposal show evidence of a qualitative and quantitative content-driven assessment of grades K-12 teacher professional learning needs with respect to math and/or science? Is the current status of student achievement in math and/or science for the targeted grades analyzed and disaggregated by gender, ethnicity, socio-economic, ELL & disability status in table form? Are other demographic student data analyzed and used to develop the plan?

Guiding Questions: Are planned activities supported by current research on effective professional learning practices and mathematics or science learning? Is that research cited in the proposal? Does the proposal show evidence of a qualitative and quantitative content-driven assessment of grades K-12 teacher professional learning needs with respect to math and/or science? Is the current status of student achievement in math and/or science for the targeted grades analyzed and disaggregated by gender, ethnicity, socio-economic, ELL & disability status in table form? Are other demographic student data analyzed and used to develop the plan?

Guiding Questions: Are planned activities supported by current research on effective professional learning practices and mathematics or science learning? Is that research cited in the proposal? Does the proposal show evidence of a qualitative and quantitative content-driven assessment of grades K-12 teacher professional learning needs with respect to math and/or science? Is the current status of student achievement in math and/or science for the targeted grades analyzed and disaggregated by gender, ethnicity, socio-economic, ELL & disability status in table form? Are other demographic student data analyzed and used to develop the plan?

________________________

12________________________

12________________________

12________________________

12

Exceeds Standard (4 Pts. each)

Meets Standard (2-3 Pts. each)

Below Standard (0-1 Pt. each) K-5 6-8 9-12 Avg.

Includes current scientifically-based research from multiple sources on effective professional learning practices

Includes sufficient research on effective professional learning practicesEvidence that the applying LEA meets qualification criteria

Limited research data on effective professional learning practices is providedLacks evidence of qualification criteria. (automatic disqualification)

46

Evidence of content-driven qualitative and quantitative assessment of current teacher professional learning needs

Evidence of content-driven assessment of current teacher professional learning needs

Limited evidence of content-driven teacher needs assessment

Student achievement data in math/science and other data for targeted grades is disaggregated in table form and analyzed in the narrative

Student achievement data in math and/or science is included and disaggregated for the targeted grades in table form

Limited student achievement data in math and/or science is included for the targeted grades

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion C: Alignment of Goals and Objectives with Professional Learning Needs

(18 Possible Points)

Criterion C: Alignment of Goals and Objectives with Professional Learning Needs

(18 Possible Points)

Criterion C: Alignment of Goals and Objectives with Professional Learning Needs

(18 Possible Points)Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Does the proposal focus on increased teacher content knowledge, ability to analyze student thinking, and make better instructional decisions? Are the program goals sufficiently ambitious, yet reasonable? Are the proposed objectives aligned to applicable Oklahoma Academic Standards (OAS), and do they include measurable outcomes correlated to the identified needs? Do proposed strategies and activities address the identified needs? Are the objectives attainable and are they measurable? Are they SMART?

Guiding Questions: Does the proposal focus on increased teacher content knowledge, ability to analyze student thinking, and make better instructional decisions? Are the program goals sufficiently ambitious, yet reasonable? Are the proposed objectives aligned to applicable Oklahoma Academic Standards (OAS), and do they include measurable outcomes correlated to the identified needs? Do proposed strategies and activities address the identified needs? Are the objectives attainable and are they measurable? Are they SMART?

Guiding Questions: Does the proposal focus on increased teacher content knowledge, ability to analyze student thinking, and make better instructional decisions? Are the program goals sufficiently ambitious, yet reasonable? Are the proposed objectives aligned to applicable Oklahoma Academic Standards (OAS), and do they include measurable outcomes correlated to the identified needs? Do proposed strategies and activities address the identified needs? Are the objectives attainable and are they measurable? Are they SMART?

________________________

18________________________

18________________________

18________________________

18

Exceeds Standard (5-6 Pts. each)

Meets Standard (2-4 Pts. each)

Below Standard (0-1 Pt. each)

K-5 6-8 9-12 Avg.

Goals/objectives are specifically linked to the identified professional learning needs and aligned to applicable OAS

Goals/objectives are generally linked to the identified professional learning needs and loosely aligned to OAS

Goals and objectives are not correlated with the needs assessment or aligned to specific OAS

Objectives are all incremental, measurable, and can be evaluated both qualitatively and quantitatively

Objective are incremental, somewhat measurable and would be difficult to evaluate both qualitatively and quantitatively

Objectives are not incremental and measurable both qualitatively and quantitatively

Goals/objectives are very realistic in scope and well defined related to the resources available

Goals and objectives are somewhat realistic in scope and well defined related to the resources available

Goals and objectives are not realistic in scope related to the resources available

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

47

Criterion D: Efficacy of Plan (32 Possible Points)Criterion D: Efficacy of Plan (32 Possible Points)Criterion D: Efficacy of Plan (32 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Are planned activities rigorous, content-focused, and supported by research on effective professional learning practices? Are planned activities likely to increase teachers’ content knowledge (TCK), strengthen ability to analyze student thinking, and further develop ability to make effective instructional decisions and improve classroom practice? Are planned activities likely to facilitate improved student achievement in math and/or science? Are meaningful follow-up sessions planned for teachers?

Guiding Questions: Are planned activities rigorous, content-focused, and supported by research on effective professional learning practices? Are planned activities likely to increase teachers’ content knowledge (TCK), strengthen ability to analyze student thinking, and further develop ability to make effective instructional decisions and improve classroom practice? Are planned activities likely to facilitate improved student achievement in math and/or science? Are meaningful follow-up sessions planned for teachers?

Guiding Questions: Are planned activities rigorous, content-focused, and supported by research on effective professional learning practices? Are planned activities likely to increase teachers’ content knowledge (TCK), strengthen ability to analyze student thinking, and further develop ability to make effective instructional decisions and improve classroom practice? Are planned activities likely to facilitate improved student achievement in math and/or science? Are meaningful follow-up sessions planned for teachers?

________________________

32________________________

32________________________

32________________________

32

Exceeds Standard (6-8 Pts. each)

Meets Standard (3-5 Pts. each)

Below Standard (0-2 Pts. each)

K-5 6-8 9-12 Avg.

Planned sessions are ambitious enough to create substantial change in TCK and improvement in classroom practice

Planned activities are somewhat ambitious enough to create substantial and positive change in TCK and improvement in classroom practice

Planned activities are weak and have limited potential of creating substantial and positive change in TCK and improvement in classroom practice

Clear and detailed description of how and when the partnership will carry out more than 80 hours of training/teacher

Acceptable description of how and when the partnership will carry out at least 80 hours of training/teacher

Limited description of how and when the partnership will carry out sessions; Lacks evidence of 80 hours/teacher

Clear and detailed evidence that the planned sessions match the specific professional learning needs and project goals

General description of how the planned sessions match the specific professional learning needs and project goals

Limited or no correlation is described between the planned sessions, the needs assessment, and project goals

Includes evidence to recruit, serve, and retain teacher cohort groups from schools of greatest academic and instructional need

Includes evidence to recruit, serve, and retain teacher cohort groups from schools of academic/instructional need

Lacks evidence of a thorough plan to recruit, serve, and retain teacher cohort groups from schools with academic and/or instructional need

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion E: Evaluation and Accountability Plan (20 Possible Points)Criterion E: Evaluation and Accountability Plan (20 Possible Points)Criterion E: Evaluation and Accountability Plan (20 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Does the evaluation plan measure the impact of the project on the specified objectives? Does the plan include personnel with expertise to implement the evaluation design? Are the procedures for measuring identified outcomes clearly identified? Will the procedures yield both qualitative and quantitative results? Will the evaluation contribute to continuous improvement? Are both pretest and posttest measures included in the plan? Does the plan employ a quasi-experimental or experimental design to measure impact of professional development on teacher content growth?

Guiding Questions: Does the evaluation plan measure the impact of the project on the specified objectives? Does the plan include personnel with expertise to implement the evaluation design? Are the procedures for measuring identified outcomes clearly identified? Will the procedures yield both qualitative and quantitative results? Will the evaluation contribute to continuous improvement? Are both pretest and posttest measures included in the plan? Does the plan employ a quasi-experimental or experimental design to measure impact of professional development on teacher content growth?

Guiding Questions: Does the evaluation plan measure the impact of the project on the specified objectives? Does the plan include personnel with expertise to implement the evaluation design? Are the procedures for measuring identified outcomes clearly identified? Will the procedures yield both qualitative and quantitative results? Will the evaluation contribute to continuous improvement? Are both pretest and posttest measures included in the plan? Does the plan employ a quasi-experimental or experimental design to measure impact of professional development on teacher content growth?

________________________

20________________________

20________________________

20________________________

20

Exceeds Standard (4-5 Pts. each) Meets Standard (2-3 Pts. each)

Below Standard (0-1 Pt. each)

K-5 6-8 9-12 Avg.

Plan includes external evaluator and valid/reliable instruments to yield quantitative & qualitative, formative & summative indicators of goal attainment

Plan utilizes evaluator and instruments to yield quantitative and qualitative, formative and summative indicators of project goal attainment

Plan lacks intention/evidence to use an evaluator and/or instruments that will yield quantitative and qualitative indicators of project’s progress

48

Specifies multiple measures and pre- and post-test procedures to show differences in TCK

Specifies pre and post procedures to show differences in TCK

Lacks a plan to use procedures to show meaningful differences in teacher effectiveness

Includes instruments and clear method to determine impact on classroom instruction and student achievement

Specifies ways to measure impact on classroom instruction and student achievement

Weak articulation of how the partnership will measure impact on classroom instruction and student achievement

Plan articulates how activities will help the MSP Program build rigorous, cumulative, reproducible, usable findings Plan employs a quasi-experimental or an experimental design using comparison or control groups to measure growth

Specifies how learning gained from the planned activities will be utilized by the partnership and the MSP Program

Lacks specification of how the learning gained from the planned activities will be utilized by the partnership

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion F: Budget and Cost Effectiveness (9 Possible Points)Criterion F: Budget and Cost Effectiveness (9 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Is the requested budget appropriate to achieve the proposed outcomes with regard to the number of teachers impacted by the proposed activities? Does the budget narrative present detailed justification for all expenses? Do budgeted items directly relate to the project goals and objectives? Will the primary partners i.e. the high-need LEA and IHE receive and use most of the budget?

Guiding Questions: Is the requested budget appropriate to achieve the proposed outcomes with regard to the number of teachers impacted by the proposed activities? Does the budget narrative present detailed justification for all expenses? Do budgeted items directly relate to the project goals and objectives? Will the primary partners i.e. the high-need LEA and IHE receive and use most of the budget?

________________________

9________________________

9________________________

9________________________

9

Meets Standard (2-3 Pts. each) Below Standard (0-1 Pt. each) K-5 6-8 9-12 Avg.

A budget is included for each of the designated partners that supports the scope and requirements of the project and provides detail and summary for the project; Budget narratives clearly delineate cost and details concerning expenditures

Provides insufficient detail for each partner and/or does not support the scope and requirements of the project or provide adequate detail and summary for the project Budget narrative does not include a cost breakdown or includes expenditures not clearly related to the project

The amount included in each budget category is commensurate with the services or goods proposed, and the overall cost of the project is commensurate with the professional development provided and the number of teachers served

The amount included in each budget category is not commensurate with the services or goods proposed, or the overall cost of the project is not commensurate with the professional development provided and the number of teachers served

The budget includes approximately 8% for an evaluation, funds key staff to participate in state MSP meetings and regional US Dept. of ED-MSP meetings;Items budgeted are appropriate and acceptable uses of funding; Indirect costs do not exceed 8%;Program cost/teacher/hour is calculated and explained

The budget is well under the suggested 8% for an evaluation or funds for key staff to participate in MSP meetings; Some items budgeted are inappropriate or disallowable uses of funding; Indirect costs exceed 8%; Cost/teacher/hour is not calculated and/or explained

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

49

OSDE Priority Scoring Points (9 Possible Bonus Points)Points

Awarded

Schools and Teachers with Greatest Need & Alignment with Other Strategic Initiatives: (1-3 Priority Points)Exceptionally clear and specific description is provided of the partnership’s plan to recruit, serve, and retain a cohort group of teachers exclusively from schools of greatest academic/instructional need; Clear definition and justification is provided for the determination of teachers/schools with greatest academic/instructional need. Proposal clearly explains how this work aligns with the institution’s strategic plan for systemic initiatives. Concentrated efforts based on need to create systemic change are evident.

3

Meaningful Administrator Participation: (1-3 Priority Points)Proposal includes convincing evidence that building-level administrators will consistently participate in a meaningful way in training sessions. Clear definition is provided for “consistent and meaningful participation.”

3

Creative, Innovative Approach: (1-3 Priority Points)Proposal includes innovative approach for the delivery of instruction and or in the formation of partnerships.

3

Reviewer Comments (please cite pages where evidence is found in proposal):

Total: __________

Scoring CategoryPossible

PointsAwarded

Points

Criterion A: Commitment and Capacity of Partnership 9

Criterion B: Demonstration of Need and Research Base 12

Criterion C: Alignment of Goals/Objectives with Professional Learning Needs 18

Criterion D: Efficacy of Plan 32

Criterion E: Evaluation and Accountability Plan 20

Criterion F: Budget and Cost Effectiveness 9

OSDE Priority Scoring Points 9

Final Score:109

Total: __________

Reviewer’s Funding Recommendations:Reviewer’s Funding Recommendations:Reviewer’s Funding Recommendations:

☐I recommend funding this proposal at a full/

modified level.Recommended Award:

Comments:

Comments:

Comments:

☐I recommend funding this proposal only if

resources allow.Recommended Award:

50

Comments:

Comments:

Comments:

☐ I do not recommend funding this proposal. I do not recommend funding this proposal.

Comments:

Comments:

Comments:

51

Appendix D: Scoring Rubric for Applicants Having Recieved Prior Funding

Scoring Rubric for MSP 2014-2015 Repeat Applicant Proposals

Criterion A: Effectiveness of Prior State Support (11 Possible Points)Criterion A: Effectiveness of Prior State Support (11 Possible Points)Criterion A: Effectiveness of Prior State Support (11 Possible Points)Criterion A: Effectiveness of Prior State Support (11 Possible Points) Points Awarded

Guiding Questions: Realizing that the current cycle has not ended and final results are not available: Does the repeat project’s proposal abstract clearly describe the goals and objectives of its funded proposal? Does it delineate how the project budget was spent during 2013-2014? Does it include the number of teachers it intended to serve (as evidenced in the funded proposal) as well as the number it actually served? Does it effectively describe progress towards goals through a thorough description of the work that was performed and evaluated? Is compelling justification provided to explain any unintended results or challenging situations faced by the partnership?

Guiding Questions: Realizing that the current cycle has not ended and final results are not available: Does the repeat project’s proposal abstract clearly describe the goals and objectives of its funded proposal? Does it delineate how the project budget was spent during 2013-2014? Does it include the number of teachers it intended to serve (as evidenced in the funded proposal) as well as the number it actually served? Does it effectively describe progress towards goals through a thorough description of the work that was performed and evaluated? Is compelling justification provided to explain any unintended results or challenging situations faced by the partnership?

Guiding Questions: Realizing that the current cycle has not ended and final results are not available: Does the repeat project’s proposal abstract clearly describe the goals and objectives of its funded proposal? Does it delineate how the project budget was spent during 2013-2014? Does it include the number of teachers it intended to serve (as evidenced in the funded proposal) as well as the number it actually served? Does it effectively describe progress towards goals through a thorough description of the work that was performed and evaluated? Is compelling justification provided to explain any unintended results or challenging situations faced by the partnership?

Guiding Questions: Realizing that the current cycle has not ended and final results are not available: Does the repeat project’s proposal abstract clearly describe the goals and objectives of its funded proposal? Does it delineate how the project budget was spent during 2013-2014? Does it include the number of teachers it intended to serve (as evidenced in the funded proposal) as well as the number it actually served? Does it effectively describe progress towards goals through a thorough description of the work that was performed and evaluated? Is compelling justification provided to explain any unintended results or challenging situations faced by the partnership?

________________________

11

Exceeds Standard (2 Pts.)Strong evidence that prior project worked with more teachers than intended according to its funded proposal

Meets Standard (1 Pt.)Evidence that prior project worked with as many or nearly as many teachers as it originally intended; or Provides acceptable explanation of why project did not work with intended number of teachers

Meets Standard (1 Pt.)Evidence that prior project worked with as many or nearly as many teachers as it originally intended; or Provides acceptable explanation of why project did not work with intended number of teachers

Below Standard (0 Pts.)Evidence that prior project worked with significantly fewer teachers than intended; or Lacks evidence that prior project worked with intended number of teachers as stated in its funded proposal

Exceeds Standard (2 Pts.)Evidence that prior project used most or all of its allotted budgetEvidence that budget was spent effectively and appropriately to meet teacher needs

Meets Standard (1 Pt.)Evidence that prior project used the majority of its allotted budgetEvidence that budget was spent appropriately on teacher needs

Meets Standard (1 Pt.)Evidence that prior project used the majority of its allotted budgetEvidence that budget was spent appropriately on teacher needs

Below Standard (0 Pts.)Lacks evidence that prior project spent its allotted budge effectively and appropriately.

Exceeds Standard (2 Pts.)Reliable quantitative and qualitative evidence that prior project work resulted in substantial gains in teacher content knowledge (TCK)

Meets Standard (1 Pt.)Quantitative and qualitative evidence that prior project work resulted in gains in teacher content knowledge (TCK)

Meets Standard (1 Pt.)Quantitative and qualitative evidence that prior project work resulted in gains in teacher content knowledge (TCK)

Below Standard (0 Pts.)Lacks evidence that prior project work resulted in gains in teacher content knowledge (TCK)

Exceeds Standard (2 Pts.)Compelling quantitative and qualitative evidence that prior project completed proposed work and met goals and objectives

Meets Standard (1 Pt.)Clear evidence that prior project completed proposed work and met goals and objectives; orProvides acceptable justification of why prior project was not able to meet goals and objectives

Meets Standard (1 Pt.)Clear evidence that prior project completed proposed work and met goals and objectives; orProvides acceptable justification of why prior project was not able to meet goals and objectives

Below Standard (0 Pts.)Lacks evidence that prior project met goals and objectives; orLacks narrative evidence justifying why prior project did not meet its goals and objectives

Exceeds Standard (2 Pts.)Clear and compelling description of how prior project intends to use new funding to inform or build upon previous successes and lessons learned

Meets Standard (1 Pt.)Acceptable description of how prior project generally intends to use new funding to inform or build upon previous successes and lessons learned

Meets Standard (1 Pt.)Acceptable description of how prior project generally intends to use new funding to inform or build upon previous successes and lessons learned

Below Standard (0 Pts.)Lacks narrative explanation of how prior project intends to use new funding to inform or build upon previous successes and lessons learned

52

Meets Standard (1 Pt.)Clear and detailed description of how all partnership members worked collaboratively toward meeting goals and objectives; or Provides justifiable explanation for unintended partnership challenges and description of project modifications to adjust for those challenges

Meets Standard (1 Pt.)Clear and detailed description of how all partnership members worked collaboratively toward meeting goals and objectives; or Provides justifiable explanation for unintended partnership challenges and description of project modifications to adjust for those challenges

Below Standard (0 Pts.)Lacks description of how all partnership members worked collaboratively towards meeting goals and objectives; orLacks justifiable explanation for unintended partnership challenges and description of project modifications to adjust for thos challenges

Below Standard (0 Pts.)Lacks description of how all partnership members worked collaboratively towards meeting goals and objectives; orLacks justifiable explanation for unintended partnership challenges and description of project modifications to adjust for thos challenges

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Criterion B: Commitment and Capacity of Partnership (9 Possible Points)Criterion B: Commitment and Capacity of Partnership (9 Possible Points)Criterion B: Commitment and Capacity of Partnership (9 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Does the project management team have the expertise to implement and sustain a math and/or science professional learning program? Do individuals who planned the project represent the primary partners i.e. LEA and IHEs? Is there evidence that mathematicians, scientists, and/or engineers are playing major roles in the design and delivery of the proposed program? Are the roles of all partners clearly identified? Does the work plan engage all partners in meaningful ways? Is there evidence that the partners share goals, responsibilities, and accountability for the proposed work? Does the governance structure describe communication, decision-making, and fiscal responsibilities among the project partners?

Guiding Questions: Does the project management team have the expertise to implement and sustain a math and/or science professional learning program? Do individuals who planned the project represent the primary partners i.e. LEA and IHEs? Is there evidence that mathematicians, scientists, and/or engineers are playing major roles in the design and delivery of the proposed program? Are the roles of all partners clearly identified? Does the work plan engage all partners in meaningful ways? Is there evidence that the partners share goals, responsibilities, and accountability for the proposed work? Does the governance structure describe communication, decision-making, and fiscal responsibilities among the project partners?

Guiding Questions: Does the project management team have the expertise to implement and sustain a math and/or science professional learning program? Do individuals who planned the project represent the primary partners i.e. LEA and IHEs? Is there evidence that mathematicians, scientists, and/or engineers are playing major roles in the design and delivery of the proposed program? Are the roles of all partners clearly identified? Does the work plan engage all partners in meaningful ways? Is there evidence that the partners share goals, responsibilities, and accountability for the proposed work? Does the governance structure describe communication, decision-making, and fiscal responsibilities among the project partners?

________________________

9________________________

9________________________

9________________________

9

Exceeds Standard (3 Pts. each) Meets Standard (2 Pts. each) Below Standard (1 Pt. each) K-5 6-8 9-12 Avg.

Strong evidence of the number and quality of staff from the primary partners to carry out the proposed activitiesQualifications are provided for key partners’ staff and appear to be exceedingly strong

Adequate number and quality of staff from the primary partners to carry out the proposed activitiesQualifications of key partners’ staff are described and appear to be acceptable

Little evidence of the number and quality of staff from the primary partners to carry out the proposed activitiesQualifications of key partners’ staff are described but appear to be limited

Shows long term commitment of partnersInstitutional resources are given in detail

Shows commitment of partners;Institutional resources are given acceptably

Shows somewhat limited commitment of partnersInstitutional resources are given but without detail

Project is likely to impact a high percentage (>50%) of teachers in need

Project is likely to impact an acceptable percentage (25%-50%) of teachers in need

Project is likely to impact a limited percentage (<25%) of teachers

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion C: Demonstration of Need and Research Base (12 Possible Points)Criterion C: Demonstration of Need and Research Base (12 Possible Points)Criterion C: Demonstration of Need and Research Base (12 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

53

Guiding Questions: Are planned activities supported by current research on effective professional learning practices and mathematics or science learning? Is that research cited in the proposal? Does the proposal show evidence of a qualitative and quantitative content-driven assessment of grades K-12 teacher professional learning needs with respect to math and/or science? Is the current status of student achievement in math and/or science for the targeted grades analyzed and disaggregated by gender, ethnicity, socio-economic, ELL & disability status in table form? Are other demographic student data analyzed and used to develop the plan?

Guiding Questions: Are planned activities supported by current research on effective professional learning practices and mathematics or science learning? Is that research cited in the proposal? Does the proposal show evidence of a qualitative and quantitative content-driven assessment of grades K-12 teacher professional learning needs with respect to math and/or science? Is the current status of student achievement in math and/or science for the targeted grades analyzed and disaggregated by gender, ethnicity, socio-economic, ELL & disability status in table form? Are other demographic student data analyzed and used to develop the plan?

Guiding Questions: Are planned activities supported by current research on effective professional learning practices and mathematics or science learning? Is that research cited in the proposal? Does the proposal show evidence of a qualitative and quantitative content-driven assessment of grades K-12 teacher professional learning needs with respect to math and/or science? Is the current status of student achievement in math and/or science for the targeted grades analyzed and disaggregated by gender, ethnicity, socio-economic, ELL & disability status in table form? Are other demographic student data analyzed and used to develop the plan?

________________________

12________________________

12________________________

12________________________

12

Exceeds Standard (4 Pts. each)

Meets Standard (2-3 Pts. each)

Below Standard (0-1 Pt. each) K-5 6-8 9-12 Avg.

Includes current scientifically-based research from multiple sources on effective professional learning practices

Includes sufficient research on effective professional learning practicesEvidence that the applying LEA meets qualification criteria

Limited research data on effective professional learning practices is providedLacks evidence of qualification criteria. (automatic disqualification)

Evidence of content-driven qualitative and quantitative assessment of current teacher professional learning needs

Evidence of content-driven assessment of current teacher professional learning needs

Limited evidence of content-driven teacher needs assessment

Student achievement data in math/science and other data for targeted grades is disaggregated in table form and analyzed in the narrative

Student achievement data in math and/or science is included and disaggregated for the targeted grades in table form

Limited student achievement data in math and/or science is included for the targeted grades

Reviewer Comments:Reviewer Comments:Reviewer Comments:Reviewer Comments:Reviewer Comments:Reviewer Comments:Reviewer Comments:

Criterion D: Alignment of Goals and Objectives with Professional Learning Needs

(15 Possible Points)

Criterion D: Alignment of Goals and Objectives with Professional Learning Needs

(15 Possible Points)

Criterion D: Alignment of Goals and Objectives with Professional Learning Needs

(15 Possible Points)Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Does the proposal focus on increased teacher content knowledge, ability to analyze student thinking, and make better instructional decisions? Are the program goals sufficiently ambitious, yet reasonable? Are the proposed objectives aligned to applicable Oklahoma Academic Standards (OAS), and do they include measurable outcomes correlated to the identified needs? Do proposed strategies and activities address the identified needs? Are the objectives attainable and are they measurable? Are they SMART?

Guiding Questions: Does the proposal focus on increased teacher content knowledge, ability to analyze student thinking, and make better instructional decisions? Are the program goals sufficiently ambitious, yet reasonable? Are the proposed objectives aligned to applicable Oklahoma Academic Standards (OAS), and do they include measurable outcomes correlated to the identified needs? Do proposed strategies and activities address the identified needs? Are the objectives attainable and are they measurable? Are they SMART?

Guiding Questions: Does the proposal focus on increased teacher content knowledge, ability to analyze student thinking, and make better instructional decisions? Are the program goals sufficiently ambitious, yet reasonable? Are the proposed objectives aligned to applicable Oklahoma Academic Standards (OAS), and do they include measurable outcomes correlated to the identified needs? Do proposed strategies and activities address the identified needs? Are the objectives attainable and are they measurable? Are they SMART?

________________________

15________________________

15________________________

15________________________

15

Exceeds Standard (4-5 Pts. each)

Meets Standard (2-3 Pts. each)

Below Standard (0-1 Pt. each)

K-5 6-8 9-12 Avg.

Goals/objectives are specifically linked to the identified professional learning needs and aligned to applicable OAS

Goals/objectives are generally linked to the identified professional learning needs and loosely aligned to OAS

Goals and objectives are not correlated with the needs assessment or aligned to specific OAS

54

Objectives are all incremental, measurable, and can be evaluated both qualitatively and quantitatively

Objective are incremental, somewhat measurable and would be difficult to evaluate both qualitatively and quantitatively

Objectives are not incremental and measurable both qualitatively and quantitatively

Goals/objectives are very realistic in scope and well defined related to the resources available

Goals and objectives are somewhat realistic in scope and well defined related to the resources available

Goals and objectives are not realistic in scope related to the resources available

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Reviewer Comments:

Criterion E: Efficacy of Plan (28 Possible Points)Criterion E: Efficacy of Plan (28 Possible Points)Criterion E: Efficacy of Plan (28 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Are planned activities rigorous, content-focused, and supported by research on effective professional learning practices? Are planned activities likely to increase teachers’ content knowledge (TCK), strengthen ability to analyze student thinking, and further develop ability to make effective instructional decisions and improve classroom practice? Are planned activities likely to facilitate improved student achievement in math and/or science? Are meaningful follow-up sessions planned for teachers?

Guiding Questions: Are planned activities rigorous, content-focused, and supported by research on effective professional learning practices? Are planned activities likely to increase teachers’ content knowledge (TCK), strengthen ability to analyze student thinking, and further develop ability to make effective instructional decisions and improve classroom practice? Are planned activities likely to facilitate improved student achievement in math and/or science? Are meaningful follow-up sessions planned for teachers?

Guiding Questions: Are planned activities rigorous, content-focused, and supported by research on effective professional learning practices? Are planned activities likely to increase teachers’ content knowledge (TCK), strengthen ability to analyze student thinking, and further develop ability to make effective instructional decisions and improve classroom practice? Are planned activities likely to facilitate improved student achievement in math and/or science? Are meaningful follow-up sessions planned for teachers?

________________________

28________________________

28________________________

28________________________

28

Exceeds Standard (5-7 Pts. each)

Meets Standard (3-4 Pts. each)

Below Standard (0-2 Pts. each)

K-5 6-8 9-12 Avg.

Planned sessions are ambitious enough to create substantial change in TCK and improvement in classroom practice

Planned activities are somewhat ambitious enough to create substantial and positive change in TCK and improvement in classroom practice

Planned activities are weak and have limited potential of creating substantial and positive change in TCK and improvement in classroom practice

Clear and detailed description of how and when the partnership will carry out more than 80 hours of training/teacher/year

Acceptable description of how and when the partnership will carry out at least 80 hours of training/teacher/year

Limited description of how and when the partnership will carry out sessions; Lacks evidence of 80 hours/teacher/year

Clear and detailed evidence that the planned sessions match the specific professional learning needs and project goals

General description of how the planned sessions match the specific professional learning needs and project goals

Limited or no correlation is described between the planned sessions, the needs assessment, and project goals

Includes evidence to recruit, serve, and retain teacher cohort groups from schools of greatest academic and instructional need

Includes evidence to recruit, serve, and retain teacher cohort groups from schools of academic/instructional need

Lacks evidence of a thorough plan to recruit, serve, and retain teacher cohort groups from schools with academic and/or instructional need

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion F: Evaluation and Accountability Plan (16 Possible Points)Criterion F: Evaluation and Accountability Plan (16 Possible Points)Criterion F: Evaluation and Accountability Plan (16 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

55

Guiding Questions: Does the evaluation plan measure the impact of the project on the specified objectives? Does the plan include personnel with expertise to implement the evaluation design? Are the procedures for measuring identified outcomes clearly identified? Will the procedures yield both qualitative and quantitative results? Will the evaluation contribute to continuous improvement? Are both pretest and posttest measures included in the plan? Does the plan employ a quasi-experimental or experimental design to measure impact of professional development on teacher content growth?

Guiding Questions: Does the evaluation plan measure the impact of the project on the specified objectives? Does the plan include personnel with expertise to implement the evaluation design? Are the procedures for measuring identified outcomes clearly identified? Will the procedures yield both qualitative and quantitative results? Will the evaluation contribute to continuous improvement? Are both pretest and posttest measures included in the plan? Does the plan employ a quasi-experimental or experimental design to measure impact of professional development on teacher content growth?

Guiding Questions: Does the evaluation plan measure the impact of the project on the specified objectives? Does the plan include personnel with expertise to implement the evaluation design? Are the procedures for measuring identified outcomes clearly identified? Will the procedures yield both qualitative and quantitative results? Will the evaluation contribute to continuous improvement? Are both pretest and posttest measures included in the plan? Does the plan employ a quasi-experimental or experimental design to measure impact of professional development on teacher content growth?

________________________

16________________________

16________________________

16________________________

16

Exceeds Standard (4-5 Pts. each) Meets Standard (2-3 Pts. each)

Below Standard (0-1 Pt. each)

K-5 6-8 9-12 Avg.

Plan includes external evaluator and valid/reliable instruments to yield quantitative & qualitative, formative & summative indicators of goal attainment

Plan utilizes evaluator and instruments to yield quantitative and qualitative, formative and summative indicators of project goal attainment

Plan lacks intention/evidence to use an evaluator and/or instruments that will yield quantitative and qualitative indicators of project’s progress

Specifies multiple measures and pre- and post-test procedures to show differences in TCK

Specifies pre and post procedures to show differences in TCK

Lacks a plan to use procedures to show meaningful differences in teacher effectiveness

Includes instruments and clear method to determine impact on classroom instruction and student achievement

Specifies ways to measure impact on classroom instruction and student achievement

Weak articulation of how the partnership will measure impact on classroom instruction and student achievement

Plan articulates how activities will help the MSP Program build rigorous, cumulative, reproducible, usable findings Plan employs a quasi-experimental or an experimental design using comparison or control groups to measure growth

Specifies how learning gained from the planned activities will be utilized by the partnership and the MSP Program

Lacks specification of how the learning gained from the planned activities will be utilized by the partnership

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

Criterion G: Budget and Cost Effectiveness (9 Possible Points)Criterion G: Budget and Cost Effectiveness (9 Possible Points) Points AwardedPoints AwardedPoints AwardedPoints Awarded

Guiding Questions: Is the requested budget appropriate to achieve the proposed outcomes with regard to the number of teachers impacted by the proposed activities? Does the budget narrative present detailed justification for all expenses? Do budgeted items directly relate to the project goals and objectives? Will the primary partners i.e. the high-need LEA and IHE receive and use most of the budget?

Guiding Questions: Is the requested budget appropriate to achieve the proposed outcomes with regard to the number of teachers impacted by the proposed activities? Does the budget narrative present detailed justification for all expenses? Do budgeted items directly relate to the project goals and objectives? Will the primary partners i.e. the high-need LEA and IHE receive and use most of the budget?

________________________

9________________________

9________________________

9________________________

9

Meets Standard (2-3 Pts. each) Below Standard (0-1 Pt. each) K-5 6-8 9-12 Avg.

A budget is included for each of the designated partners that supports the scope and requirements of the project and provides detail and summary for the project; Budget narratives clearly delineate cost and details concerning expenditures

Provides insufficient detail for each partner and/or does not support the scope and requirements of the project or provide adequate detail and summary for the project Budget narrative does not include a cost breakdown or includes expenditures not clearly related to the project

56

The amount included in each budget category is commensurate with the services or goods proposed, and the overall cost of the project is commensurate with the professional development provided and the number of teachers served

The amount included in each budget category is not commensurate with the services or goods proposed, or the overall cost of the project is not commensurate with the professional development provided and the number of teachers served

The budget includes approximately 8% for an evaluation, funds key staff to participate in state MSP meetings and regional US Dept. of ED-MSP meetings;Items budgeted are appropriate and acceptable uses of funding; Indirect costs do not exceed 8%;Program cost/teacher/hour is calculated and explained

The budget is well under the suggested 8% for an evaluation or funds for key staff to participate in MSP meetings; Some items budgeted are inappropriate or disallowable uses of funding; Indirect costs exceed 8%; Cost/teacher/hour is not calculated and/or explained

Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments: Reviewer Comments:

OSDE Priority Scoring Points (9 Possible Bonus Points)Points

Awarded

Schools and Teachers with Greatest Need & Alignment with Other Strategic Initiatives: (1-3 Priority Points)Exceptionally clear and specific description is provided of the partnership’s plan to recruit, serve, and retain a cohort group of teachers exclusively from schools of greatest academic/instructional need; Clear definition and justification is provided for the determination of teachers/schools with greatest academic/instructional need. Proposal clearly explains how this work aligns with the institution’s strategic plan for systemic initiatives. Concentrated efforts based on need to create systemic change are evident.

3

Meaningful Administrator Participation: (1-3 Priority Points)Proposal includes convincing evidence that building-level administrators will consistently participate in a meaningful way in training sessions. Clear definition is provided for “consistent and meaningful participation.”

3

Creative, Innovative Approach: (1-3 Priority Points)Proposal includes innovative approach for the delivery of instruction and or in the formation of partnerships.

3

Reviewer Comments (please cite pages where evidence is found in proposal):

Total: __________

Scoring CategoryPossible

PointsAwarded

Points

Criterion A: Effectiveness of Prior State Support 11

Criterion B: Commitment and Capacity of Partnership 9

Criterion C: Demonstration of Need and Research Base 12

Criterion D: Alignment of Goals/Objectives with Professional Learning Needs 15

Criterion E: Efficacy of Plan 28

57

Criterion F: Evaluation and Accountability Plan 16

Criterion G: Budget and Cost Effectiveness 9

OSDE Priority Scoring Points 9

Final Score:109

Total: __________

Reviewer’s Funding Recommendations:Reviewer’s Funding Recommendations:Reviewer’s Funding Recommendations:

☐I recommend funding this proposal at a full/

modified level.Recommended Award:

Comments:

Comments:

Comments:

☐I recommend funding this proposal only if

resources allow.Recommended Award:

Comments:

Comments:

Comments:

☐ I do not recommend funding this proposal. I do not recommend funding this proposal.

Comments:

Comments:

Comments:

58

Appendix E: Professional Development Standards for the TEACHER

Learning Communities Standard 1: Professional development that improves the learning of all students organizes adults into learning communities whose goals are aligned with those of the school and district.

Objective 1.1: The teacher meets regularly with colleagues during the school day to plan instruction.

Objective 1.2: The teacher aligns collaborative work with school improvement goals. Objective 1.3: The teacher participates in learning teams, some of whose membership extends

beyond the school. Leadership

Standard 2: Professional development that improves the learning of all students requires skillful school and district leaders who guide continuous instructional improvement.

Objective 2.1: The teacher participates in instructional leadership development experiences. Objective 2.2: The teacher serves in a variety of instructional leadership roles. Objective 2.3: The teacher contributes to the planning of school-based professional learning. Objective 2.4: The teacher articulates the intended results of professional development

programs on teacher practice.Objective 2.5: The teacher advocates for support of professional development. Objective 2.6: The teacher articulates the benefits of professional learning.

Resources

Standard 3: Professional development that improves the learning of all students requires resources to support adult learning and collaboration.

Objective 3.1: The teacher participates in professional development during the workday. Objective 3.2: The teacher accesses funds to support learning priorities. Objective 3.3: The teacher receives external and internal support related to learning priorities.

Data-Driven

Standard 4: Professional development that improves the learning of all students uses disaggregated student data to determine adult learning priorities, monitor progress, and help sustain continuous improvement.

Objective 4.1: The teacher analyzes disaggregated student data to identify adult learning priorities at the classroom, school, and district levels.

Objective 4.2: The teacher analyzes a variety of disaggregated data to identify learning need of professionals.

Objective 4.3: The teacher works with colleagues to use disaggregated data to establish professional learning goals.

Objective 4.4: The teacher analyzes relevant student data in order to monitor and revise school and classroom improvement strategies.

Evaluation

Standard 5: Professional development that improves the learning of all students uses multiple sources of information to guide improvement and demonstrate its impact.

Objective 5.1: The teacher contributes a variety of data to evaluate the impact of professional development.

Objective 5.2: The teacher collects and analyzes classroom data to determine the impact of professional development.

Research-Based

Standard 6: Professional development that improves the learning of all students prepares educators to apply research to decision making.

59

Objective 6.1: The teacher uses educational research when making instructional decisions. Design

Standard 7: Professional development that improves the learning of all students uses learning strategies appropriate to the intended goal.

Objective 7.1: The teacher participates in a variety of appropriate professional development designs aligns with expected improvement outcomes.

Objective 7.2: The teacher participates in long-term and in-depth professional learning. Objective 7.3: The teacher implements new classroom practices as a result of follow-up

support. Objective 7.4: The teachers uses technology as a component of professional learning when

appropriate. Learning

Standard 8: Professional development that improves the learning of all students applies knowledge about human learning and change.

Objective 8.1: The teacher participates in professional development that mirrors expected instructional methods.

Objective 8.2: The teacher participates in professional learning that impacts depth of understanding.

Objective 8.3: The teacher participates in a variety of professional development experiences appropriate to career stage.

Objective 8.4: The teacher engages in professional development that considers participant concerns about new practices.

Collaboration

Standard 9: Professional development that improves the learning of all students provides educators with the knowledge and skills to collaborate.

Objective 9.1: The teacher participates in a school culture that is characterized by collegiality and shared responsibility.

Objective 9.2: The teacher develops knowledge about effective group process. Objective 9.3: The teacher collaborates successfully with colleagues. Objective 9.4: The teacher uses effective conflict management skills with colleagues. Objective 9.5: The teacher uses technology to support collegial interactions.

Equity

Standard 10: Professional development that improves the learning of all students prepares educators to understand and appreciate all students, create safe, orderly, and supportive learning environments, and hold high expectations for their academic achievement.

Objective 10.1: The teacher analyzes the impact of attitude, background, culture, and social class on the teaching process.

Objective 10.2: The teacher develops skills that communicate high expectations for each student.

Objective 10.3: The teacher establishes a learning environment that is emotionally and physically safe.

Objective 10.4: The teacher demonstrates respect and appreciation for students and families and for their cultural backgrounds.

Quality Teaching

Standard 11: Professional development that improves the learning of all students deepens educators' content knowledge, provides them with research-based instructional strategies to assist students in meeting rigorous academic standards, and prepares them to use various types of classroom assessments appropriately.

60

Objective 11.1: The teacher demonstrates a deep understanding of subject matter that helps students to meet rigorous standards.

Objective 11.2: The teacher uses appropriate instructional strategies that help students meet rigorous standards.

Objective 11.3: The teacher uses various classroom assessment strategies to monitor student progress toward meeting standards.

Family Involvement

Standard 12: Professional development that improves the learning of all students provides educators with knowledge and skills to involve families and other stakeholders appropriately.

Objective 12.1: The teacher develops partnerships with families and other community stakeholders.

Objective 12.2: The teacher implements strategies to increase family and caregiver involvement.

Objective 12.3: The teacher uses technology to increase communication between school and home about student learning.

61

Appendix F: DefinitionsThe following definitions are based on the definitions included in the No Child Left Behind Act of 2001. A. Professional Development: The term “professional development” means instructional

activities that: (1) Are based on scientifically based research and aligned to the Oklahoma Academic

Standards, student academic achievement standards, and assessment; (2) Improve and increase teachers’ knowledge of the academic subjects they teach; (3) Enable teachers to become highly qualified; (4) Are sustained, intensive, and classroom-focused in order to have a positive and lasting

impact on classroom instruction and the teacher’s performance in the classroom; and (5) Are not one-day or short-term workshops or conferences.

B. Scientifically Based Research: The term “scientifically based research” means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs and includes research that: (1) Employs systematic, empirical methods that draw on observation or experiment and

involve rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;

(2) Relies on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators;

(3) Is evaluated using experimental or quasi-experimental designs in which individuals, entities, programs, or activities are assigned to different conditions, with appropriate controls to evaluate the effects of the condition of interest and with a preference for random-assignment experiments or other designs to the extent that those designs contain within-condition or across-condition controls;

(4) Ensures that experimental studies are presented in sufficient detail and clarity to allow for replication or, at minimum, to offer the opportunity to build systematically on their findings; and

(5) Has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review.

C. Partnerships: The Mathematics and Science Partnership (MSP) projects are intended to enhance the capacity local teachers to enact curricula reforms that produce higher student achievement in mathematics and science. A partnership between local school districts and institutions of higher education’s science, technology, engineering, and mathematics (STEM) faculty is the conduit used to reach these goals and is required in these projects.

62