Developing State and Local Report Cards L. Condelli, M. Corley NRS Institute on Developing State and Local Report Cards for Adult Education American Institutes

Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley

NRS Institute onDeveloping State and Local Report

Cards for Adult Education

American Institutes for ResearchMiami, Florida

February 28-March 2, 2007


Goals of the InstituteGoals of the Institute

By the end of the training, participants will be able to

Discuss the purposes of, and audiences for, report cards for state and local adult ed programs;

Identify essential elements of a report card; Distinguish between effective and ineffective

report cards; Use a template to develop state and local report

cards; Create a rubric for evaluating performance; and Develop a dissemination plan.

02


Agenda for Wednesday, Feb. Agenda for Wednesday, Feb. 2828

Lessons Learned: States’ Experiences with Report Cards

A Word about Quality: States’ Plans to Improve Quality

Step 1: Define Purpose of and Audience for State and Local Report Cards

Step 2: Select Data Elements/Measures for State and Local Report Cards

Step 3: Select Evaluative Criteria or Rubric Step 4: Design and Format Report Cards State Planning Time and Status Reports Introducing the User Guide and Report Card

Templates Wrap-up and Evaluation for Day 1

03


Agenda for Thursday, March Agenda for Thursday, March 11

Navigating the Template A Trial Run with a Hypothetical State Q&A about the Template State Teams Build Local Report Cards Progress Checks and Support State Teams Demonstrate Local

Report Cards Parking Lot/Discussion, Wrap-up, and

Evaluation for Day 2

04


Agenda for Friday, March 2Agenda for Friday, March 2 State Teams Build State Report

Cards Progress Checks and Support State Teams Demonstrate State

Report Cards Step 5: Disseminate and Promote

Program Improvement State Teams Develop Dissemination

Plans Next Steps, Wrap-up, and Evaluation

05


Why Report Cards?Why Report Cards?

Accountability—Measure Adequate Yearly Progress

Program Improvement

Inform and Advocate for Program

06


Questions for ConsiderationQuestions for Consideration What would a report card on

hospitals tell you?

What about one on surgeons?

What about one on airlines?

What about one on stocks?

What would you do with the information?

07


What is a Report Card?What is a Report Card?

A concise presentation of data and other information about a school or program that assesses performance

The focus on evaluating performance is what makes report cards a unique type of accountability report.

08


Characteristics of Characteristics of EffectiveEffective Report Cards Report Cards

Focus on outcomes and other data that reflect program quality;

Provide a basis of comparison for evaluating these data; and

Present contextual data or interpretive information that aid interpretation and promote understanding.

09


Characteristics of Characteristics of IneffectiveIneffective Report Cards Report Cards

Provide poor indicators of quality

Do not support program improvement

Are not informative

10


Three Primary Factors of Three Primary Factors of IneffectiveIneffective Report Cards Report Cards

1. Wrong measures

2. Insufficient measures

3. Unreliable or low quality data

11


A Word about A Word about AggregatedAggregated v. v. DisaggregatedDisaggregated Data Data

We weighed each of you as you entered this room, so we know that the average weight of everyone in this room is 195 lbs. And we posted that average on the wall.

Is this good or bad news?What does this tell us about our entire population?What does it tell us about males v. females?Is this a weight loss group or a steroid users’ support group?

12


Help us understand patterns of success and failure in the student population;

Help us separate the “whys” from the “whines”

Caution: When we report information on student learning as a mean, it tells us about as much as the weight sign on our wall.

Disaggregated Data…Disaggregated Data…

13


NCLB Requirements for NCLB Requirements for State Report CardsState Report Cards

Concise; understandable and uniform format Info on student achievement, both

aggregated and disaggregated Comparison between actual achievement

levels and the state’s annual goals/objectives

Most recent trends in student achievement Performance of local agencies re: AYP with

names of schools identified for school improvement

Professional qualifications of teachers Description of state’s accountability system

14


NCLB Requirements for NCLB Requirements for LocalLocal DistrictDistrict Report Cards Report Cards

The same data required for the state report card as applied to the local education agency

Performance of students of local district compared to performance of students in the state as a whole

Performance of local agencies re: AYP with names of schools identified for school improvement

15


DifferencesDifferences Between Uses of Between Uses ofState and Local Report CardsState and Local Report Cards

Local: Include a broader range of outcome

measures to evaluate program More suited for program improvement

efforts Use evaluative standard such as program

past performance, performance of similar programs, or local performance standard

State: Use a more limited range of outcome

measures Use evaluative standards such as state past

performance, state performance standards, or national averages16


NRS Outcome Measures: NRS Outcome Measures:

The The CenterpieceCenterpiece of the Report of the Report

CardCard

Educational gain Receipt of a secondary credential Entered and retained employment Entry into postsecondary education

17


Other Measures?Other Measures?

Student attendance and persistence data

Student and employer satisfaction with program

Learner accomplishments Number of hours of instruction

offered Average class size Teacher qualifications

18


Contextual Data that Contextual Data that Aid InterpretationAid Interpretation

Student demographics

Socioeconomic data about the community

Per student cost

19


Basis for EvaluationBasis for Evaluation

To evaluate program quality, must have a standard or basis of comparison

Compare students and programs to each other and to fixed standards

Examples: letter grades, past performance, average state performance

20


Five Steps to Developing a Five Steps to Developing a Report Card for Adult Report Card for Adult

EducationEducation

1.1. DefineDefine purpose and audience

2.2. SelectSelect measures

3.3. SelectSelect evaluative criteria or rubric

4.4. DesignDesign and format

5.5. DisseminateDisseminate and promote program improvement

21


1. Define Purpose and 1. Define Purpose and AudienceAudience

Purpose: evaluation, program improvement, or information?

State or local report card? Audience: local program staff, state staff,

funding agencies, legislators, general public? Which information to show/not show? Is this

different for difference audiences? In what areas are your programs doing

well/doing poorly? Why? What messages do you want to send to

audience and what do you want them to do about it?

22


2. Select Measures2. Select MeasuresOther Outcome Measures

Instruction Measures

Program and Teacher Measures

Student Measures

State or local test performance measures, such as test score gains

Other employment measures, e.g., better job

Family literacy measures

Amount and type of classes and instruction offered

Average hours of attendance

Budget and average cost per student

Support services provided

Number of teachers and qualifications

Average class sizes

Literacy skills upon program entry

Demographics of service area (e.g., % without H.S. diploma;unemployment rate)

23


Matrix of Audience, Report Card Matrix of Audience, Report Card Purposes, Measures and Purposes, Measures and

ComparisonsComparisonsRpt Card Purposes

Main Audiences Measures and Comparison

Evaluate Program Quality

State and local staff General public Legislators Funding agencies Partner agencies

Basic outcome measures compared to performance standards, past performances, state or national averages

Program Improvement

TeachersLocal program staff

Broader outcome measures compared to similar programs, local performance standards, or past performance

Inform and Advocate

General publicLegislatorsFundersPartner agencies

Outcomes with comparisons of similar programs’ outcomes & with contextual data about need and community conditions

24


3. Select Evaluative Criteria or 3. Select Evaluative Criteria or RubricRubric

Evaluative Criteria Use

Program Past Performance

Demonstrate or monitor continuous improvement

Performance Targets

Uniform standard for programs

State Averages Compare local performance against others in the state

Other Local Program Performance

Compare local performance and account for variations among programs

25


What is a Rubric?What is a Rubric?

A scheme for classifying products or behaviors into categories that vary along a continuum

A vehicle for describing varying levels of quality, from excellent to poor, or from meets expectations to unacceptable

Examples of rubrics we are familiar with: letter grades, or the GED essay scoring guide

26


Advantages of Scoring Advantages of Scoring RubricsRubrics

Clarify expectations about the characteristics of quality programs;

Set standards for program performance; Provide an objective measure for examining

and evaluating programs; Can provide formative feedback to programs; Can be used by programs for self-assessment

and improvement; Can lead to shared standards among staff

and the public about what makes a good program.

27


Writing Descriptions of Writing Descriptions of RubricsRubrics

Keep criteria or indicators specific, objective, and value neutral;

Describe what the levels of quality look like, without using judgmental language;

Be sure indicators define progress along a continuum from lowest to highest quality.

28


Seven Steps to Seven Steps to Developing a RubricDeveloping a Rubric

1.1. Identify Identify the measure for the rubric and a possible range of responses.

2.2. IdentifyIdentify the highest possible range of scores to define the top category.

3.3. Define Define an unacceptable level.4.4. DefineDefine the lowest level of acceptable

performance.5.5. DefineDefine an intermediate level, between

the top and the lowest.6.6. LabelLabel the categories.

7.7. Test Test and refine.

29


4. Design and Format4. Design and Format

Design PrinciplesDesign Principles Use of high quality graphics and photos Simple, short, succinct

Data PresentationData Presentation Avoid needless data breakdowns and

disaggregation Simplify data for your audience Be careful with percentages, base

numbers, and response rates

30


Graphic Displays Should…Graphic Displays Should… Show the data; Induce the viewer to think about the

substance, not the methodology or graphic design;

Avoid distorting what the data have to say; Make large data sets coherent; Encourage the eye to compare different pieces

of data; Reveal data at several layers of detail, from

a broad overview to the fine structure; Be closely integrated with the statistical

and verbal descriptions of a data set.

31


Some Some Do’sDo’s and a and a Don’tDon’t DoDo make report card text short and

easy to read. DoDo make student performance

prominent… but report more than test scores.

DoDo be cautious about assigning labels to programs.

Don’tDon’t overdo displays of demographic data.

32


Questions to Ask When Questions to Ask When FramingFraming thethe MessageMessage

What does this audience segment need to know?

What are the few key points that best illustrate what you want this audience to know about your program?

What data best supports your message?

Remember: How you say it does matter.

33


Don’tDon’t Use Use KillerKiller Words WordsDifficult Death Obligation Wrong

Failure Decisions

Fail Bad

Deal Cost Sell Taxes

Liability Worry Lose Hard

Contract Buy Complicated

Dangerous

34


Disaggregated data Standards-based tests Alternative assessment Performance index Chi-square, p-value, theta,

coefficient Others?

Don’tDon’t Use Use JargonJargon

35


The 30-3-30 FormulaThe 30-3-30 Formula

Audience time spent perusing a product 30 seconds: most people. 3 minutes: a smaller segment that will

read headings, subheads, illustrations, opening and summary statements

30 minutes: the smallest segment that will read the whole product

Guess which of the above applies to OVAE staff??

36


Does Your Message Pass Does Your Message Pass the the Joe Six-Pack TestJoe Six-Pack Test??

(i.e., Does your neighbor understand the message?)

If not, radically oversimplify.

If you don’t, the media will, and will inevitably get it wrong.

37


User’s Guide for User’s Guide for Creating Report CardsCreating Report Cards

User’s Guide for Creating and Formatting State and Local Report Cards

38


User’s Guide for User’s Guide for Creating Report CardsCreating Report Cards

39


PurposePurpose:: What is the purpose of this report card? e.g., training or monitoring?

AudienceAudience:: Who will review this report card? What message should this audience take away from the report card about program quality?

MeasuresMeasures:: What combination of measures will provide locals with a clear message, given the purpose?

Getting Started – Getting Started – Create a Local Report CardCreate a Local Report Card

40



RubricRubric:: What criteria/rubric will indicate the quality of the outcomes or measure progress that programs are achieving?

State Standard

AExemplary

B Meets Standard

C Below Standard

FUnacceptable

Level Completions

ABE = 40% > 40% 37% - 40% 32% - 36% < 32%

ASE = 35% > 35% 32% - 35% 29% - 31% < 29%

ESL = 32% > 32% 29% - 32% 26% - 28% < 26%

41


DesignDesign:: Which graphs, tables, text, and pictures, will most effectively communicate the intended message?

DisseminationDissemination:: Begin to consider how the report card will

be disseminated to the audience. How will that affect the design, including how data might be interpreted by various audiences?


42


Begin Begin by selecting a Local Report Card template

Use Use examples as guides: Report Card Templates tab contains sample

NRS report card templates for guidance; Other Report Cards tab contains current

Adult Ed and K-12 examples for ideas. Report outReport out –

Share the Local Report Card draft; Highlight purpose, audience,

measures, rubric, and design choices.

Action!Action!

43


5. Disseminate and 5. Disseminate and Promote Program Promote Program

ImprovementImprovement Disseminate

1.1. DevelopDevelop a distribution plan. Vehicle: meetings, mailings, Web site, etc. Audience: legislators, funding agencies, public, etc.

2.2. MakeMake the report cards accessible.3.3. ProvideProvide guidance in interpreting the data.4.4. UseUse credible messengers in the community.5.5. AvoidAvoid defensiveness.6.6. MakeMake the report card easy to find on Web sites or

easy to access by mail or phone.7.7. DistributeDistribute to legislators and funding agencies.8.8. CultivateCultivate relationships with the media, if

permitted.

44


Questions to Answer Questions to Answer BeforeBefore Disseminating Report CardsDisseminating Report Cards

Who gets the report card (state or local)? By what means (distribution channel)? By what date? How will you evaluate the effectiveness…

Of the report card? Did it serve the purpose you planned or hoped for?

Of the distribution channel? Did the message reach the intended audience effectively and efficiently?

45


Questions to Answer Questions to Answer BeforeBefore Disseminating Report CardsDisseminating Report Cards

How will the report cards be packaged? How will staff handle incoming

requests for the report cards? Will there be a change for postage?

Will the report cards be made available in pdf format on your Web page?

Who will maintain the Web page?

46


Questions to Answer Questions to Answer BeforeBefore Disseminating Report CardsDisseminating Report Cards Will you arrange for media coverage

upon the report card’s release? If so, how? What media sources? Will you work through your

system’s/district’s public relations/public affairs officer?

After the report cards are “out there,” will you have a system to accept input from the audience on the quality of the reports and the dissemination system?

47


Distribution ChannelsDistribution Channels

Direct mail Email Web Pages (Your home page) Newspapers and other print media TV/radio Board meetings and other

appropriate community meetings Credible spokespersons

(representatives of the intended audience segment)

48


The The Bottom LineBottom Line in in DisseminationDissemination

Do not rely on any single messenger or distribution method.

The most effective communications strategy uses multiple messengers and channels.

49


5. Disseminate and 5. Disseminate and Promote Program Improvement Promote Program Improvement

(Cont.)(Cont.)

Promote Program Improvement Promote use and understanding Opportunities for technical

assistance Recognition for good

performance

50


Questions for Questions for ConsiderationConsideration

As you look at your data… What do these data seem to tell

you? What do they not tell you? What else do you need/want to

know? What good news is here for you to

celebrate? What needs for program

improvement arise from these data?

51


Next Steps?Next Steps?

How can we help you?

www.nrsweb.org

52


Best Wishes in Your Best Wishes in Your Continuing Quest for Continuing Quest for

Quality…..Quality…..

Thank You and Good Luck!

53

Documents

Developing State and Local Report Cards L. Condelli, M. Corley NRS Institute on Developing State and Local Report Cards for Adult Education American Institutes