Supplement 1 MeasuringOutcomesandImp

8/13/2019 Supplement 1 MeasuringOutcomesandImp

1/27

Supplement 1: Measuring

Outcomes and Impacts


2/27

Sequence number

987654321

16

14

12

10

8

6

4

2

0

Probable and

Durable

Probable and

Non-Durable

Improbable

Regression to Mean

Improbable Constant

Change

Improbale Non-Linear

Change

Policy Intervention

Patterns of Causality in Time Series


3/27

Conditions Required to

Make Causal Inferences Condition X precedes condition Y in time

X O

Condition X is correlated with condition Y rx.y > 0 (+/-)

Conditions other than X do not affect

condition Yr x.y = rx.y|z


4/27


5/27

Strengths of Quasi-Experimental Designs

Activity theory of causation

Recognition of systemic complexity

Financial, political, and ethical feasibility

Rarity of true experiments

Availability of resources

(www.economagic.com, www.fedstats.gov,

www.census.gov, www.eurostat, StatistickiGodisnjak/ Bilten)
http://www.economagic.com/http://www.fedstats.gov/http://www.census.gov/http://www.eurostat/http://www.eurostat/http://www.census.gov/http://www.fedstats.gov/http://www.economagic.com/


6/27

Extended Time Series

I O1 O2 O3 O4 O5 O6 O7


7/27

Interrupted Time Series

I O1 O2 O3 X O4 O5 O6 O7


8/27

Control Series

I O1 O2 O3 X O4 O5 O6 O7

II O1 O2 O3 ~X O4 O5 O6 O7


9/27

Problems with Interrupted Time Series

Incremental diffusion of programs with no sharpcutting points

Multiple programs operating at same time

Lack of detailed knowledge of program activities Insufficient observations in time series

Unknown time intervals due to delays in

implementing programs Multiple rival explanations of outcomes


10/27

Interrupted Time-Series AnalysisHelps Detect Causality

Sequence number

987654321

16

14

12

10

8

6

4

2

0

Probable and

Durable

Probable and

Non-Durable

Improbable

Regression to Mean

Improbable Constant

Change

Improbale Non-Linear

Change

Policy Intervention


11/27

Some Outcome Indicators HousingArea per person (square meters)

Average Life Expectancy Quality Adjusted Life Years

Persons Below Poverty Line

Income Distribution (Gini Index)

Air Pollution Index (parts per million) Lead Concentration Index (blood concentration)

Persons in Mental Hospitals

Average Test Scores

Sales or Market Share

Votes Cast for Candidates

Foreign Direct Investment in MKD

Number of newly licensed foreign companies


12/27

The Odds Ratio Measures Effect Size

Example--It is believed that more highly educated voters tend to votefor Democratic candidates in the U.S. Here is a sample of voters

who voted in the 1992 Presidential Election. How would a policy of

producing more Masters and Ph.D.graduates affect the outcome of

elections?

Clinton Bush and Perot

Less than

Masters

Masters or

Ph.D. Degree

797

(0.48)

82

(0.42)

857

(0.52)

111

(0.58)

P

1-Q

1-P

Q

1,654

(1.0)

193

(1.0)

P / 1-P = 0.92

Q / 1-Q = 1.38

ODDS RATIO =

1.38 / 0.92 = 1.5


13/27

The Standardized Mean Difference

Measures Effect SizeExampleBetween 1987 ands 1989 the maximum speedlimit in 40 of the 50 states of the U.S. was increased from55mph to 65mph. The paired t-test, which involves achange in means from t0to t+1(Note: Observations in anytime series are notindependent), was used to test the null

hypothesis that there is no statistically significant difference(p = 0.05) between traffic fatalities before (1987) and after(1989) the speed limit was raised to 65 mph in 40 states.The speed limit was kept at 55 mph for 10 states. Whatdoes the following test show about the effects of removingthe old (55mph) policy?

texp= mean fatality rate after the policy - mean fatality ratebefore the policy / pooled standard deviation

= -0.23 / 0.35 = -0.66

tcon = mean fatality rate after the policymean fatality ratebefore the policy / pooled standard deviation

= -0.07 / 0.10 = -0.70


14/27

Guidelines for Interpreting

Standardized Effect Sizes 0.80-0.99 strong

0.60-0.79 moderate to strong

0.40-0.59 moderate 0.20-0.39 weak to moderate

0.00-0.19 negligible to weak

NOTE: The practical significance of an effect size

depends on the social costs of being wrong.


15/27

Other Measures of Effect Size

Identical Units of Measure. Benefits and costs in constant value of acurrency, unemployment rates, percent of budget variance,performance appraisal scale.

Established Norms. Dietary intake of vitamins compared with minimum(RDA) required daily amount, international test scores, percentabove poverty line, percent below a living wage.

Average Effect Sizes.Average correlations in political science andsociology range from r = 0.20 to r = 0.30. Average internalconsistency reliabilities for mental health inventories, placementexaminations, and other instruments involving high risk of beingwrong are r > 0.95.

Coefficient of Variation. The standard deviation divided by the mean

(CV= s/ m) . This is the percent variability divided by the mean. Astandard deviation, s,of 16 with a mean of 100 is the same as astandard deviation, s,of 96 with a mean of 600. The variability oflarge municipal budgets can be compared with smaller ones.


16/27

Pooled t-Test. The outcome mean after the intervention subtractedfrom the outcome mean before the intervention, divided by the

pooled standard deviation. NOTE: The observations before and afterthe intervention are not independent and therefore the pooed t-testmust be used.

x2x1/ sqrt [Sp (1/n1) + (1/n2)]

Standard (z) Scores.An individual score subtracted from the mean ofthe distribution divided by the standard deviation.

z =x - m / s. An individualscore of 116 from a distribution with amean of 100 and a standard deviation of 16 is the same as anindividual score of 348 from a distribution with a mean of 300 and a

standard deviation of 48. Individual scores measured with twodifferent scales, or from two different distributions, can be directlycompared.


17/27


18/27

Interrupted Time-Series

With Three Observations

YEAR

19751974197319721971

56000

54000

52000

50000

48000

46000

44000

42000

56000

54000

52000

50000

48000

46000

44000

42000


19/27

Extended Time-Series

With Interruption

Fig. 8.2. Connecticut traffic fatalities, 1951-59

YEAR

.195919581957195619551954195319521951.

340

320

300

280

260

240

220


20/27

Extended Time-Series

With Interruption

YEAR

2000

1998

1996

1994

1992

1990

1988

1986

1984

1982

1980

1978

1976

1974

1972

1970

1968

1966

60000

50000

40000

30000


21/27

Fig. 8.3. Connecticut and control states traffic fatalities, 1951-59

YEAR

.195919581957195619551954195319521951.

340

320

300

280

260

240

220

Control States

Connecticut

Control Series With Interruption


22/27

Transforms: natural log, difference (1)

YEAR

2000

1998

1996

1994

1992

1990

1988

1986

1984

1982

1980

1978

1976

1974

1972

1970

1968

.2

.1

0.0

-.1

-.2

Fatalities

Economic Index

Changes in Fatalities Per Mile

Correlated with Economic Factors


23/27

Control Series With Interruption:

Fatality Rates in Europe and the US

YEAR

76757473727170

28

26

24

22

20

18

16

US

EURCON

EUREXP


24/27

Annual Changes in Fatality Rate

and Miles Driven, 1913-2000

Year

1996

1990

1984

1978

1972

1966

1960

1954

1948

1942

1936

1930

1924

1918

1912

1906

.4

.2

0.0

-.2

-.4

Billion Miles Driven

Traffic Fatalities


25/27

Group Problem

Examine the extended time-series graphsshowing the observed fatality rate, the predictedfatality rate, and European Commission targetfor 2010.

1. Explain how interrupted time-series analysis might resultin a different predicted fatality rate. Is the observedfatality rate a valid predictor of fatalities in future?

2. Explain how control-series analysis might change theCommissions 2010 target fatality rate. Is the targetrealistic?


26/27

ForecastEU Fatality Rate by 2010

0

10

20

30

40

1980 1985 1990 1995 2000 2005 2010

fatalitiesperb

illionveh-km

f atality rate model ETSC forecast


27/27

0

10000

20000

30000

40000

50000

60000

70000

80000

1970 1975 1980 1985 1990 1995 2000 2005 2010

Fatalities

perye

ar

European Commission Proposed Target:

50% Reduction Between 2000 and 2010

Documents

Supplement 1 MeasuringOutcomesandImp