Evaluating Data Reliability Validity –bias Reliability Consistency of Measurement –Changes in...

Evaluating Data

• Reliability

• Validity– bias

Reliability

• Consistency of Measurement– Changes in Definition– Changes in Record keeping– Changes in public understanding

• Sampling Error– Other Sampling issues

Measurement Validity

• Does it measure what it is supposed to?

• Bias: under- or over- measure for particular groups

• EXAMPLE: SAT tests, poverty measures,

1. After each state’s scores for each category werestandardized (by dividing the observed value for eachstate by the highest value for all states, to create ascore ranging from zero to one), they were weightedaccording to the relative degree of influence andimportance of the position.

State representatives weregiven a weight of 1.0; state senators, 1.25; statewideexecutive elected officials (except governors), 1.5;governors, 1.75. The resulting scores were added tocreate a total score for each state. Scores on thecomposite index were then used to rank the states forwomen’s representation in elected office. 

POVERTY MEASURES

• FDR: "one-third of a nation ill-housed, ill-clad, ill-nourished"

• US Poverty Rate: absolute standard

• Ireland, Smeeding “relative poverty rate”

• Canada: Low-income limit, based on disposable income

• World Bank: Consumption levels below $1 or $2 per day.

Absolute Poverty Standard: World Bank

Number of Poor Families, by Family Type, 1973-2001(in millions)

2.8

0.6

3.5

0

1

2

3

4

5

1973 1977 1981 1985 1989 1993 1997 2001

single mother

-- single father

married couple

What is Poverty?

• In 1963-1964, Molly Orshansky based poverty thresholds on the "thrifty food plan,“

• Thrifty food budget (family of 4) = $1,033• Families spent 1/3 of their budget on food.• 1963 Poverty level = $3,100• Subsequent years, adjusted for CPI

inflation index

Poverty thresholds

Problems with definition of Poverty

• Wealth and debt not counted• Unusual family expenditures not

counted.• Regional living costs• Families now spend only 1/6th

income on food• MEASURING INCOME……….

Measuring Income• money income before taxes, including

wages, salaries, interest, dividends, self-employment income, welfare payments (TANF), unemployment insurance, and social security payments.

• Poverty Report from March Current Population Survey, 50,000 households

Problems with the Definition of Family Income

• Income not counted: Food Stamps, Medicaid & Medicare, Public Housing

• Persons not counted: the institutionalized, Army barracks, unrelated individuals under 15

Problems defining “Family”

• Must be related

• Family defined in March,BUT Income, in previous year

• Unrelated children individual counted as neither poor nor non-poor.

Overcounting the poor?

• Inflation adjustment overestimates inflation.

• Food stamps not counted as income.

• Other in-kind benefits (which have increased overtime) not counted. (Housing, Medical).

• Earned Income Tax Credit

• Under reporting of income.

Undercounting the Poor?

• Food now takes 1/6 of typical budget.

• High regional living costs

• Unusual family living costs

• Increased FICA taxes

• Sample under-represents poor