46
June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and as guidance to help you complete the Validation peer review process.

June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

Embed Size (px)

Citation preview

Page 1: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

June 11, 2010

Development Panel Monitor Orientation

Investing in Innovation (i3) Orientation

Note: These slides are intended for internal use only and as guidance to help you complete the Validation peer review process.

Page 2: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

2

What is the role of a PANEL MONITOR?

2

• Contact reviewers to schedule panel calls• Assist reviewers with any questions• Identify any conflicts of interest• Facilitate panel calls for all 20 applications on each panel

• Make sure reviewers enter comments/scores in e-Reader promptly

• Promptly send feedback on written comments/scores to reviewers and notify reviewers when scores are to be SUBMITTED in final form

Panel Monitor

Page 3: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

3

Investing in Innovation (i3) Fund Summary

3

Purpose

Funding

Applicants

To provide competitive grants to applicants with a record of improving student achievement and attainment in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on:•Improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates

$650 million to be obligated by September 30, 2010

Eligible applicants are: (1)Local educational agencies (LEAs) (2)Nonprofit organizations in partnership with (a) one or more LEAs or (b) a consortium of schools

Page 4: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

4

Types of Awards Available Under i3

4

i3

EstimatedFunding Available

Up to $5MM/award Up to $30MM/award Up to $50MM/award

Evidence Required

Reasonable – research findings or hypotheses, including related research or theories in education and other sectors

Moderate – either high internal validity and medium external validity, or vice versa

Strong – both high internal validity and high external validity

Scaling Required

Able to further develop and scale

Able to be scaled to the regional or state level

Able to be scaled to the national, regional, or state level

Page 5: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

5 5

MUST

MUST

Eligible Applicant:

LEA

Eligible Applicant:

LEA

•Demonstrate that it: (a) significantly closed achievement gaps between groups of students or demonstrated success in significantly increasing academic achievement for all groups of students, and(b) made significant improvement in other areas

Establish partnerships with private sectorSecure commitment for required 20% private sector matchMeet the evidence requirement for the type of grant for which it has applied

•Demonstrate that it: (a) significantly closed achievement gaps between groups of students or demonstrated success in significantly increasing academic achievement for all groups of students, and(b) made significant improvement in other areas

Establish partnerships with private sectorSecure commitment for required 20% private sector matchMeet the evidence requirement for the type of grant for which it has applied

TO RECEIVE A GRANT,MUST

Page 6: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

6 6

MUST

MUST

Eligible Applicant:

Non-profits, in partnership with LEA(s) or a consortium of schools

Eligible Applicant:

Non-profits, in partnership with LEA(s) or a consortium of schools

Demonstrate that the non-profit organization has a record of significantly improving student achievement, attainment, or retention through its record of work with an LEA or schools

Secure commitment for required 20% private sector matchMeet the evidence requirement for the type of grant for which they have applied

Demonstrate that the non-profit organization has a record of significantly improving student achievement, attainment, or retention through its record of work with an LEA or schools

Secure commitment for required 20% private sector matchMeet the evidence requirement for the type of grant for which they have applied

TO RECEIVE A GRANT,MUST

Page 7: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

7

All Eligible Applicants Must Implement Practices, Strategies, or Programs for

High-Need Students

7

MUST

MUST

High-need student means a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools, who are far below grade level, who are over-age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient.

High-need student means a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools, who are far below grade level, who are over-age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient.

Page 8: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

8 8

1. Teacher and Principal Effectiveness

2. Improved Use of Data Systems

3. College- and Career-ready Standards and

High Quality Assessments

4. Improving Achievement in Persistently Low-performing

Schools

5. Early Learning(0 or 1 point)

6. College Access and Success

(0 or 1 point)

7. Serving Students with Disabilities and Limited

English Proficient Students(0 or 1 point)

8. Serving Students in Rural LEAs

(0, 1, or 2 points)

i3 Priorities

Must select one(Absolute Priority)

May select one or more(Competitive Preference

Priorities)

Page 9: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

9 9

Notes on Absolute Priority 1Innovations that Support Effective Teachers and Principals

• “…increase the number or percentages of teachers or principals who are highly effective teachers or principals or reduce the number or percentages of teachers or principals who are ineffective, especially for teachers of high-need students…”

• “…by identifying, recruiting, developing, placing, rewarding, and retaining highly effective teachers or principals (or removing ineffective teachers or principals).”

• “…teacher or principal effectiveness should be determined through an evaluation system that is rigorous, transparent, and fair; performance should be differentiated using multiple rating categories of effectiveness; multiple measures of effectiveness should be taken into account, with data on student growth as a significant factor, and the measures should be designed and developed with teacher and principal involvement.”

Two Possible Routes

Multiple Measures of Effectiveness

Multiple Methods

Page 10: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1010

Notes on Absolute Priority 2 Innovations that Improve the Use of Data

• “…(a) encourage and facilitate the evaluation, analysis, and use of student achievement or student growth data by educators, families, and other stakeholders in order to inform decision-making and improve student achievement, student growth, or teacher, principal, school, or LEA performance and productivity; or (b) enable data aggregation, analysis, and research”

• “…data must be disaggregated using the student subgroups described in section 1111(b)(3)(C)(xiii) of the ESEA…”

Two Possible Areas of Focus

Data Disaggregation

Page 11: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1111

Notes on Absolute Priority 3 Innovations that Complement the Implementation of High

Standards and High-Quality Assessments• “…standards and assessments that measure students’

progress toward college and career-readiness…”

• “…may include, but are not limited to, … • increase the success of underrepresented student

populations in academically rigorous courses and programs…;

• increase the development and use of formative assessments or interim assessments, or other performance-based tools and “metrics” that are aligned with high student content and academic achievement standards; or

• translate the standards and information from assessments into classroom practices that meet the needs of all students, including high-need students.

• “…eligible applicant must propose a project that is based on standards that are at least as rigorous as its State’s standards…”

Focus on College & Career Readiness

Range of Allowable Projects

RigorousStandards

Page 12: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1212

Notes on Absolute Priority 4 Innovations that Turn Around

Persistently Low-Performing Schools

• “Whole-school reform, including, but not limited to, comprehensive interventions to assist, augment, or replace Investing in Innovation Fund Absolute Priority 4 schools, including the school turnaround, restart, closure, and transformation models of intervention … OR …”

• “Targeted approaches to reform, including, but not limited to:

• Providing more time for students to learn core academic content by expanding or augmenting the school day, school week, or school year, or by increasing instructional time for core academic subjects

• integrating ‘‘student supports’’ into the school model to address non-academic barriers to student achievement

• creating multiple pathways for students to earn regular high school diplomas”

Projects May Choose Either

Approach

Page 13: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1313

i3 Priority 4 Schools

Under Absolute Priority 4, the Department provides funding to support strategies, practices, or programs that are designed to turn around schools that are in any of the following categories:

(a)persistently lowest-achieving schools (as defined in the final requirements for the School Improvement Grants program, see http://www2.ed.gov/programs/sif/faq.html);

(b)Title I schools that are in corrective action or restructuring under section 1116 of the Elementary and Secondary Education Act (ESEA); or

(c)secondary schools (both middle and high schools) eligible for but not receiving Title I funds that, if receiving Title I funds, would be in corrective action or restructuring under section 1116 of the ESEA.

Page 14: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

14

i3 Absolute Priorities

Applicants must address ONE Absolute Priority

No perks for addressing more than one Absolute Priority

Absolute Priority is not scored

14

Page 15: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1515

Notes on Competitive Preference Priority 5 Innovations for Improving Early Learning Outcomes

• “…improve educational outcomes for high-need students who are young children (birth through 3rd grade) by enhancing the quality of early learning programs”

• “…(a) improving young children’s school readiness (including social, emotional, and cognitive readiness) so that children are prepared for success in core academic subjects (as defined in section 9101(11) of the ESEA); (b) improving developmental milestones and standards and aligning them with appropriate outcome measures; and (c) improving alignment, collaboration, and transitions between early learning programs that serve children from birth to age three, in preschools, and in kindergarten through third grade.”

Focus on High-need Children

Projects Must Address All 3

Page 16: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1616

Notes on Competitive Preference Priority 6 Innovations that Support College Access and Success

• “… enable kindergarten through grade 12 (K–12) students, particularly high school students, to successfully prepare for, enter, and graduate from a two- or four-year college…”

• “…(a) address students’ preparedness and expectations related to college; (b) help students understand issues of college affordability and the financial aid and college application processes; and (c) provide support to students from peers and knowledgeable adults.”

Focus on College

Graduation

Projects Must Address All 3

Page 17: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1717

Notes on Competitive Preference Priority 7 Innovations to Address the Unique Learning Needs of Students

with Disabilities and Limited English Proficient Students

• “…address the unique learning needs of students with disabilities, including those who are assessed based on alternate academic achievement standards, or the linguistic and academic needs of limited English proficient students.”

• “…must provide for the implementation of

particular practices, strategies, or programs that are designed to improve academic outcomes, close achievement gaps, and increase college- and career-readiness, including increasing high school graduation rates (as defined in this notice), for students with disabilities or limited English proficient students.”

Focus on Either Student

Population

Projects That Improve Specific

Outcomes

Page 18: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

1818

Notes on Competitive Preference Priority 8 Innovations that Serve Schools in Rural LEAs

• “…focus on the unique challenges of high-need students in schools within a rural LEA…”

• “…must include practices, strategies, or programs that are designed to improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or improve teacher and principal effectiveness in one or more rural LEAs.”

Focus on Specific Locations

Projects May Address Range of Outcomes

Page 19: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

19

Scoring the Competitive Preference Priorities (CPP)

Applicants may address ONE or MORE CPPsCPP 5, 6, and 7 are scored as all or nothing (1

or 0)CPP 8 is scored with 0,1, or 2A CPP may have a score of 0 or 1 and still

have strength and weakness commentsOnly the subject matter reviewers will

score the CPPs.

19

Page 20: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

20

i3 Selection Criteria and Points

20

* Development grants will be judged in two tiers: all eligible applications will be scored on Criteria A, C, E, F, and G and the competitive preference priorities; then high-scoring applications will be scored on Criteria B and D by a different panel of reviewers.

Selection Criteria Development Validation Scale-UpA. Need for the Project and

Quality of the Project Design 25 20 15

B. Strength of Research, Significance of Effect and Magnitude of Effect

10* 15 20

C. Experience of the Eligible Applicant

25 20 15

D. Quality of the Project Evaluation

15* 15 15

E. Strategy and Capacity to Bring to Scale or to Further Develop and Bring to Scale

5 10 15

F. Sustainability 10 10 10G. Quality of the Management

Plan and Personnel 10 10 10

Total Points 100 100 100

Page 21: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

Subject Matter Reviewers will Score Criteria A, C, E, F, and G (NOT the Evidence and Evaluation Criteria)

Subject Matter Review

Page 22: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

22

Development GrantsA. Need for the Project and Quality of the Project

Design (up to 25 points)

22

The Secretary considers the need for the project and quality of the design of the proposed project. In determining the need for the project and quality of the design of the proposed project, the Secretary considers the following factors:

1)The extent to which the proposed project represents an exceptional approach to the priorities the eligible applicant is seeking to meet (i.e., addresses a largely unmet need, particularly for high-need students, and is a practice, strategy, or program that has not already been widely adopted).2)The extent to which the proposed project has a clear set of goals and an explicit strategy, with the goals, objectives, and outcomes to be achieved by the proposed project clearly specified and measurable and linked to the priorities the eligible applicant is seeking to meet.

Page 23: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

23

Development GrantsC. Experience of the Eligible Applicant (up to 25

points)

23

The Secretary considers the experience of the eligible applicant in implementing the proposed project. In determining the experience of the eligible applicant, the Secretary considers the following factors:

•The past performance of the eligible applicant in implementing projects of the size and scope proposed by the eligible applicant.

•The extent to which an eligible applicant provides information and data demonstrating that—

(a) In the case of an eligible applicant that is an LEA, the LEA has— (i) Significantly closed the achievement gaps between groups of students described in section 1111(b)(2) of the ESEA, or significantly increased student achievement for all groups of students described in such section; and (ii) Made significant improvements in other areas, such as graduation rates or increased recruitment and placement of high-quality teachers and principals, as demonstrated with meaningful data; or

(b) In the case of an eligible applicant that includes a nonprofit organization, the nonprofit organization has significantly improved student achievement, attainment, or retention through its record of work with an LEA or schools.

Page 24: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

24

Development GrantsE. Strategy and Capacity to Further Develop and Bring to Scale (up to

5 points)

24

The Secretary considers the quality of the eligible applicant’s strategy and capacity to further develop and bring to scale the proposed project. In determining the quality of the strategy and capacity to further develop and bring to scale the proposed project, the Secretary considers:

•The number of students proposed to be reached by the proposed project and the capacity of the eligible applicant and any other partners to reach the proposed number of students during the course of the grant period.

2)The eligible applicant’s capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to further develop and bring to scale the proposed practice, strategy, or program, or to work with others (including other partners) to ensure that the proposed practice, strategy, or program can be further developed and brought to scale, based on the findings of the proposed project.

•The feasibility of the proposed project to be replicated successfully, if positive results are obtained, in a variety of settings and with a variety of student populations. Evidence of this ability includes the availability of resources and expertise required for implementing the project with fidelity, and the proposed project’s evidence of relative ease of use or user satisfaction.•The eligible applicant’s estimate of the cost of the proposed project, which includes the start-up and operating costs per student per year (including indirect costs) for reaching the total number of students proposed to be served by the project. The eligible applicant must include an estimate of the costs for the eligible applicant or others (including other partners) to reach 100,000, 250,000, and 500,000 students.•The mechanisms the eligible applicant will use to broadly disseminate information on its project so as to support further development or replication.

Page 25: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

25

Development Grants F. Sustainability (up to 10 points)

25

The Secretary considers the adequacy of resources to continue the proposed project after the grant period ends. In determining the adequacy of resources for the proposed project, the Secretary considers the following factors:

•The extent to which the eligible applicant demonstrates that it has the resources, as well as the support of stakeholders (e.g., State educational agencies, teachers' unions), to operate the project beyond the Development grant.•The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the eligible applicant and any other partners at the end of the Development grant.

Page 26: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

26

Development Grants G. Quality of the Management Plan and Personnel

(up to 10 points)

26

The Secretary considers the quality of the management plan and personnel for the proposed project. In determining the quality of the management plan and personnel for the proposed project, the Secretary considers:

•The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks.

•The qualifications, including relevant training and experience, of the project director and key project personnel, especially in managing projects of the size and scope of the proposed project.

Page 27: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

DEVELOPMENT REVIEW

Review Process & Reviewer Expectations

Page 28: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

28

What is the basic i3 review process?

33

For all three grant types…

• The Department will use independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest

• Applications will be assigned to panels by absolute priority where possible

• Evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation in the Tier 2 review

• Peer reviewers will determine whether any competitive preference priority points should be added

Development Only…

• Development applications will be reviewed in a two tier process– In Tier 1, all eligible applicants will be reviewed and scored

against Selection Criteria A, C, E, F and G by three peer reviewers. Competitive Preference Points will also be added as appropriate by peer reviewers.

– Only those highest rated in Tier 1 will advance to Tier 2, where Selection Criteria B and D will be scored by two peer reviewers who are evidence and evaluation experts.

Page 29: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

29

What is expected of a Peer Reviewer?

34

Process

Behavior

• Review the entire i3 application package and FAQs

• Review and become thoroughly familiar with the selection criteria, factors, and notes

• Familiarize yourself with the e-Reader system – used for inputting scores and comments

• Participate in all scheduled conference calls

• Provide appropriate comments that justify the score awarded and are helpful to the applicant

• Revise comments as suggested by your panel monitor

• Return all forms as required to ensure payment and completion of review process

• Be available the entire review process

• Draw upon your expertise

• Maintain confidentiality throughout the review process

Preparation

Page 30: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

30

What must a Peer Reviewer really do? Receive applications and panel assignments from Synergy Identify any Conflicts of Interest, and notify Synergy or

panel monitor immediately Participate in Orientation webinar Register in and become acquainted with e-Reader Read applications (It may be helpful to you to take notes!) Write draft comments, assign scores, and submit drafts in e-

Reader before panel discussion (We recommend cutting and pasting from Word)

Receive feedback on comments from Panel Monitor Revise draft comments based on feedback as appropriate When comments/scores are acceptable to Panel Monitor,

submit final comments/scores Sign and return Technical Review Signature Forms to Synergy Remember to submit All Necessary Forms to Synergy35

Prior to Panel Discussion

After Panel Discussion

Page 31: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

31

Registering in e-Reader http://e-grants.ed.govCall the e-Grants help desk at 1-888-336-3930 to request a

PIN# for the registration processClick the “Continue” button and the “Register” button Indicate that you are an ED Employee and select the e-

Reader moduleComplete the User Registration page and SubmitA system generated password will be e-mailed to youUse the username and password to loginExisting e-Reader users should use their existing username

and password.

Help Desk Hours of Operation

Tuesday, Friday and Saturday: Available 24 hoursMonday and Thursday: 6am – midnight; Sunday: midnight – 8pm and Wednesday: midnight – 7pm

Page 32: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

32

Panel Call 101Panel calls are held at 8:30 AM, 11:30 AM, and 2:30

PM dailyReviewers/Panel Monitors have selected their

preferred time slotPanel calls are held by phone and each panel has its

own conference call number4-6 Applications should be covered on each panel call

until all 20 applications are completedPanel calls will likely last the entire 2 ½ hour period June 14-18, all panel calls should be completeReview follow up should be completed by June 25,

2010Adjustments to panel call times may be arranged

between the reviewers and panel monitorBe FLEXIBLE and expect EMERGENCIES

Page 33: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

33

Conducting a Panel CallContact reviewers no later June 11 to schedule panel calls Use the first call to go over the review process, introductions,

and answer initial questionsGo over the selection criteria and CPPs for each applicationHave reviewers discuss strengths and weaknesses for each

criterion and CPPs, if applicableUse prompt questions & reviewers’ scores to generate panel

discussionsUse the score comparison sheet to keep track of reviewers’

scores and scoring changesMake sure scores received are justified by the written

comments If needed, conduct a second panel call to further discuss

applications that have a 20 percent varianceComments should be continuously reviewed until you give

reviewers final approval to SUBMIT comments in e-Reader43

Page 34: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

34

Reviewing Application Scores Numeric scores indicate how well the applicant responded to the

selection criteria Use the entire range of points for each criterion. There is NO scoring rubric, so all factors should be evaluated equally For each of the competitive preference priorities, applicants do not earn a

point simply for addressing the priority. Reviewers must award points based on how well applicants address the requirements of the priority.

Make sure that your comments are consistent with your numeric scores If full points are awarded to a criterion, all comments should be in

“strengths” section and “no weaknesses found” should be entered in the “weaknesses” section.

If partial points are awarded to a criterion, there should be points and comments under both the “strengths” and “weaknesses” sections.

ZERO means that something is missing from a criterion and all reviewers MUST agree in order to award ZERO points to a criterion

For CPPs, ZERO means that the applicant did not provide a quality response to the priority. All reviewers DO NOT have to agree to award ZERO points on a CPP37

Page 35: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

35

Reviewing Application Comments

38

“Do’s” “Don’ts”

Application should be read and scored against the selection criteria

Comments should support the numeric score and should be based on the strengths and weaknesses of an application’s responses to the selection criteria.

Comments should be clear, concise, and objective.

Comments should be written in complete sentences

Remind reviewers to use the spelling and grammar check on all comments before they submit

Remind reviewers to include application page numbers as they are helpful in the comments

• Don’t compare applications. The should be evaluated independently

• Don’t Summarize, paraphrase or quote information presented in the application without adding comments that explain your judgment about that information

• Don’t Write one- or two-word comments to justify your scores

• Don’t Use inflammatory or derogatory language when writing comments

• Don’t ask questions of the applicant in comments

• Don’t copy and paste comments from applications and use them in other applications

Page 36: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

36

Reviewing Application Comments (cont’d)Provide feedback/edits to reviewers on their

comments promptly (next slide)

Have reviewers keep you posted on the status of their comments

Develop a system to track what you have reviewed, need to review, need from reviewers, and what TRFs are complete

Keep in mind that applicants will be receiving the reviewer comments, so review them thoroughly

Keep in mind that comments may be made public, so review them thoroughly

39

Page 37: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

37

How to Provide Feedback to Reviewers

Copy and paste reviewers comments from e-Reader into a Word doc, then use Track Changes to edit comments. Email this document to the reviewer

Use Outlook or e-Reader to email edits/feedback to reviewers. You could list edits for each application by criterion

Either way is fine, use what works best for you

Page 38: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

38

Sample Comment – #1

“The application proposes to create a professional development program for teachers that will instruct them how to use performance data from class assignments and to use it to improve their teaching. It’s an excellent idea.”

Problems: The comment simply paraphrases the application, and it does not explain why the reviewer likes the idea.

Revision: “One of the strengths of the proposed professional development program is that it gives teachers direct practice in using the newly created online repository of student assignments that can be used “in the moment” when classroom assessments identify a learning deficiency. As the application demonstrates on p. 17, this technology-based approach aligns with research showing that learning deficits can be remedied if addressed immediately.”

Why is it Better?: It explains more clearly why the reviewer believes this is a quality program, and it includes specific references to the proposal.

40

Page 39: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

39

Sample Comments – #2“This management plan seems thorough and thoughtful, with appropriate LEA representation on the advisory council. The allocation of leadership and responsibilities seems appropriate to the tasks, and the schedule laid out in the chart seems reasonable.”

Problems: At first glance, these comments seem fine, but they are not as helpful as they could be. More detail is necessary to explain what about the management plan is “thorough and thoughtful.” What characteristics of the schedule or the allocation of responsibilities is the reviewer referencing?

Partial Revision: “…It makes sense in this case to split administrative leadership of the project from the coordination of the each of the school-based implementation sites and to assign these duties to different individuals…”

Why is it Better?: The revision makes more clear what specific aspects of the leadership plan are “appropriate” and “reasonable.”41

Page 40: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

40

Sample Comments – #3

(Weakness) “The administrative employee is on contract and is not a full-time employee. Is that advisable?”

Problem: It may appear that points have been deducted based on an issue of uncertainty, and the applicant cannot respond. It’s generally best to avoid questions. No weakness has been specifically identified.

Revision: Because the administrative employee is contracted, rather than serving as a full-time employee, it is unclear whether the applicant would be able to absorb this individual’s critical functions after the grant period is over.

Why is it Better?: The revision refrains from asking a question and more clearly explains why there might be a weakness in the way this employee has been budgeted.

42

Page 41: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

41

Accessing the Technical Review Form (TRF)

Log into e-Reader, select “Reports”, select “84.396C (2010)”

Score Comparison gives you overall scores; selecting the application gives you a breakdown of scores

Reviews brings you to the TRF for each reviewer for each application

Select a TRF for a reviewer, then select Print/View Form to view or print the TRF

Page 42: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

42

Disable “Smart Quotes” in MS Word

Peer Reviewers are encouraged to write their comments is MS Word, then copy and past them into e-Reader. Before you start typing in MS Word , you should disable Smart Quotes to avoid formatting problems in e-Reader. The instructions are below:

o In MS Word on the Tools menu, click AutoCorrect. o Go to the AutoFormat As you Type tab. o Uncheck the option “Straight quotes” with “smart quotes.” o Uncheck the option Symbol characters (- -) with symbols (--).o Go to the AutoFormat tab. o Uncheck the option “Straight quotes” with “smart quotes.” o Uncheck the option Symbol characters (- -) with symbols (--). o Click the OK button.

45

Page 43: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

43

Other Important Resources

46

Investing in Innovation Fund Website: (http://www.ed.gov/programs/innovation/index.html)

Notice of Final Priorities, Requirements, Definitions, and Selection Criteria

Application Package (includes the Notice Inviting Applications)

Frequently Asked Questions Evidence Summary Table Selection Criteria Summary Table

i3 Overview (PowerPoint) i3 At-A-Glance (Quick Reference) Archived recordings of the i3 Webinars i3 Glossary

Orientation Slides Panel Monitor Orientation Handouts

Page 44: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

44

Questions

Please direct all questions to:Thelma Leenhouts

(i3 Competition Manager)

[email protected](202) 260-0223

47

Page 45: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

45

Compensation for a job well done!!

We will offer Comp time or Overtime. Each person must select only one

You may receive up to 40 hours of comp time or overtime over the two week review period

Comp time expires after 26 pays (one year)Overtime is time and a ½ if you are a grade 7 or 9Grades 11 and higher will only receive you regular

hourly pay http://www.opm.gov/oca/10tables/html/dcb_h.asp

Part time employees may receive only comp time or they may work up to 40 hours

48

Page 46: June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and

The i3 Core Team would like to thank you for all your support in helping us to complete this i3

grant competition.

THANK YOU