If you can't read please download the document
Upload
joie
View
45
Download
1
Embed Size (px)
DESCRIPTION
Planning Sample Size for Randomized Evaluations. Jed Friedman, World Bank Based on slides from Esther Duflo, J-PAL. Planning Sample Size for Randomized Evaluations. General question: How large does the sample need to be to credibly detect a given effect size? - PowerPoint PPT Presentation
Citation preview
Planning Sample Size for Randomized EvaluationsJed Friedman, World BankBased on slides from Esther Duflo, J-PAL
Planning Sample Size for Randomized EvaluationsGeneral question: How large does the sample need to be to credibly detect a given effect size? What does Credibly mean here? It means that I can be reasonably sure that the difference between the group that received the program and the group that did not is due to the programRandomization removes bias, but it does not remove noise: it works because of the law of large numbers how large much large be?
Basic set upAt the end of an experiment, we will compare the outcome of interest in the treatment and the comparison groups. We are interested in the difference: Mean in treatment - Mean in control = Effect sizeFor example: mean of the number of bed nets in villages with free distribution v. mean of the number of bed nets in villages with cost recovery
Estimation
But we do not observe the entire population, just a sample
In each village of the sample, there is a given number of bed nets. It is more or less close to the actual mean in the total population, as a function of all the other factors that affect the number of bed nets
We estimate the mean by computing the average in the sample
If we have very few villages, the averages are imprecise. When we see a difference in sample averages, we do not know whether it comes from the effect of the treatment or from something else
Estimation
The size of the sample:Can we conclude if we have one treated village and one non treated village? Can we conclude if we give IPT to one classroom and not the other? Even though we have a large class size?What matter is the effective sample size i.e. the number of treated units and control units (e.g. class rooms). What is it the unit the case of IPT given in the classroom?
The variability in the outcome we try to measure: If there are other many non-measured things that explain our outcomes, it will be harder to say whether the treatment really changed it.
When the Outcomes are Very Precise
Chart4
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
10
30
60
120
180
170
220
130
30
40
00
11
012
012
016
020
013
011
09
03
03
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet4
46.9976784087470
37.2231683185370
52.4425730771522
62.764735402632
61.9835021906622
67.331331037671
28.1641236041282
47.6581875671482
60.9502252599611
39.1329935053393
43.0979583951432
33.0956767255333
31.5308910911324
40.2237050264404
42.2649294604422
28.8206878293293
44.3207512842446
45.9595243126464
51.3485305317513
46.3450704854462
46.7300936987474
46.297594862465
63.4264155349632
49.1471554495491
48.138423507481
44.8679260342454
69.7221197595704
58.656729733593
73.7565473071744
43.4509332888434
66.6145582625670
33.8760231796342
55.3894837038552
59.0219145932593
69.1891558643692
49.154829311490
44.7620494822450
56.7513838076572
46.1867615639460
57.5761136031582
35.5581336305360
41.527624843420
34.7842900635352
46.3712298268460
49.6752080733501
50.2811702871500
46.7728399497470
71.9450157601720
32.575172907733
42.635230228243
24.22419255624
64.476700016564
37.202363627137
43.464200542843
57.577136784758
54.667117536955
58.746087587559
55.957417670356
36.281500241636
38.842614584439
56.939944796657
53.226364242453
40.601622812441
47.590521161348
51.315356712451
55.577976480756
51.387149950551
40.890387380241
68.84845914969
54.871981218555
50.722388904351
58.298411557958
58.620077096659
43.634685324744
40.768083080241
61.111887943161
37.988212524238
34.411078912734
57.11324901257
56.384061634956
72.056883608572
64.437546269664
63.039039004163
51.129603788351
50.019508661450
54.537014319855
49.744852630150
39.453249327239
32.251938490732
58.283313945958
54.442244971954
56.179061529156
52.13473185852
39.73069068640
62.381951819462
46.887868292147
41.600782323142
41.788718034542
45.710072653346
45.466384916745
Sheet1
valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60
300030003000
310031003100
322032003200
332033003300
342034003400
351135003510
362036003600
372137003710
381138003820
393039003920
402040004020
413141004120
424042004250
434043004310
442244004450
453345104551
466146304640
474247604781
4831481204881
4923491804980
5043501705061
5152512205123
5220521305275
531453305332
541454405421
554055005545
564356115687
5736570125725
5847580125847
5947590165928
6003600206026
6123610136106
6225620116213
633563096314
642364036417
650465036506
660366006614
672467006702
680268006804
692169006905
700170007000
710071007101
722172007200
730273007300
741074007400
750075007500
760376007601
770077007700
78178007800
79179007900
80080008000
81381008100
82082008200
83083008300
84184008400
85085008500
86086008600
87087008700
88088008800
89089008900
90090009000
Sheet1
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet5
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet7
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet9
66.5623453338670
56.6444306665570
58.5494923749590
53.991343672540
58.4543364917580
37.6646291139381
58.6645889294590
46.708007782471
61.5657974473621
65.0302332966650
73.1708475237730
53.8070459393541
52.5203610474530
61.4636498236610
57.6856997819582
57.9168933351583
75.7720023708761
60.3768491297602
58.5402155289591
78.3032170753783
53.6926542432543
72.7113708004732
59.8466137185600
59.3558276401594
76.101557776764
63.6263145375640
64.3705881581643
57.6487515596586
66.8332724368677
59.3136498233597
52.6392320049533
49.0246726864493
62.0370066522625
50.18662038505
79.3957930605793
58.7278442859594
44.1236785869443
56.6468521961574
69.5723180493702
56.1702997098561
53.4613583719531
46.3361608732460
63.1611534723631
58.3629095368582
65.7125362218660
53.62626113540
56.5382244227573
62.8593831303630
56.689712134571
62.1222149371621
35.2617441118350
59.3113533492593
62.4685959943620
81.1649421544810
44.7476829018451
44.4136709726440
36.9523128634370
71.7225226859720
83.7436324824840
56.1002458804560
49.9916019785500
65.479932951965
66.589880285967
51.479330674351
68.092911230168
66.602249413867
57.9254880658
44.51849023745
70.127769201170
67.634344001468
56.495148479956
53.09213762953
68.626716407969
63.468278464463
65.562787919266
80.858897187381
62.733690962563
48.93340489349
64.79632262865
47.714613780648
65.655897439366
58.434975623258
62.988713276963
63.57082399264
61.313037500961
57.280610841757
46.526336215347
81.208234102281
75.976957001876
70.488224688770
45.220028975245
48.826925800849
65.326774044165
60.735428784561
59.910119186160
61.612283995262
41.183009367741
57.432701093257
50.620631251751
50.024366525250
Sheet8
51.5621208149520
50.0357272256500
48.9833486325490
48.4957594279480
49.0737342159490
50.261218247500
50.2365482032500
48.8936906448490
45.712278178460
46.7639269016470
50.3006198313500
52.6035195333530
50.6633490557510
50.6333561942510
48.834177859490
49.5841608325501
47.7187962941483
51.8372656996526
48.36266852074812
49.22318238534918
49.60524291965017
45.28394255324522
49.54499344315013
49.8679459168503
51.1556812751514
49.8785233402500
51.9424714992521
46.7312123772470
49.8424800651500
50.9221798791510
48.0860184223480
49.1095228324490
49.9050305632500
46.5371534927470
47.1606666804470
54.3186810217540
48.3907400747480
48.4523674357480
46.8429438013470
50.9023165148510
51.4216675481510
48.3375118973480
51.1437691683510
49.0433093444490
50.6900813787510
48.3697171047480
51.3527733245510
48.9379921317490
51.3600924831510
51.2249029169510
49.4692871041490
48.5710360306490
51.5505247575520
50.9635573406510
50.7099265531510
49.8627299647500
48.9925640875490
46.2330184644460
53.93272785540
52.8293061405530
52.2297672331520
51.54124336452
54.260691639554
49.896144850
49.426372596649
51.850607986952
50.018742412150
50.53055600851
46.675869624647
49.702938566850
51.075773070651
48.649147983149
47.661871020748
53.134391590753
51.775611053752
51.643115865652
51.610105755452
48.979167230649
51.614127995752
47.81106907948
50.006043592350
48.70644842349
51.298269580751
50.509189703751
52.302144821452
50.940426616651
49.125207068649
51.751236595752
51.03287675351
51.270254870251
54.399262252354
48.263751876948
46.357983036446
50.85597548651
48.177845554948
48.779521774849
56.232876330656
50.532140802551
50.341690338250
49.40065663449
Sheet6
65.0056496548650
57.3209696691570
60.8248753147610
58.7707633409590
63.3077071748630
53.2927312329530
62.3775396585620
55.0287224237550
55.9214619594560
58.5760314303590
59.0897845237590
54.2942349662540
68.5711235442690
44.5087586436450
63.3339210859630
57.4822912918571
69.304158084690
49.7818815953501
58.874643552591
58.586890797590
69.8715190688701
64.4599050852643
63.5056245906645
65.9267858887662
59.6263682078601
59.0135643202595
68.5255305748697
68.9733293862695
56.2703191159567
57.0830049278578
51.8423782219526
57.5707351041586
58.5415433912593
60.9120776667614
52.1062362773527
68.2275801105686
58.3764564604584
47.2046100791472
65.7252736951664
56.2178367241565
51.0154292593510
62.336387297621
59.834517439600
75.5964517035760
60.7276730685610
64.807507139650
60.3975469554601
71.2473026093710
51.1998475510
64.382231964640
56.7147141433570
60.4325124792600
69.1942274594690
55.0461642584550
66.9574298323670
51.5543185125520
58.255666469580
60.0562272362600
55.0468190946550
69.6492021839700
62.5066856324630
64.557912234365
67.589892447968
67.524954526268
50.931446518551
65.59492036766
69.76300725670
55.817915987656
56.04949607656
57.632883150858
63.749742063564
69.713294275670
61.498674464561
47.508781487148
58.376456460458
61.405564944461
65.994452294566
61.66261543262
52.568850757453
62.701867742963
67.035341695967
64.696962605565
57.088693817257
51.758299984252
68.459310265668
64.098224053564
58.130306266858
55.354005477655
65.173369572765
58.921791757359
64.330254341864
61.341009010561
64.66460278465
55.584139469456
55.63872279456
57.878528574858
60.015838850260
55.218399817555
51.860850059352
63.863131041764
Sheet2
57.8872290032580
63.997341739640
41.8400590104420
47.380170953470
53.8276266423540
47.9565291142481
51.918901944520
55.9200601754561
49.5412531462502
50.89907644512
49.3804090082492
48.2460894898482
55.9029116528565
62.1690391097621
63.2923014462635
48.255666469485
46.2407628118464
56.4707228375568
49.7504664861508
51.1038764569518
48.2206873028486
47.3194485392472
38.7579906499397
55.7383158492563
52.4437190405522
43.2019409243434
49.2311541064498
37.3942942941372
39.6987173997404
46.9776922626472
48.3450106811482
47.2073965182470
40.1883984421401
51.717539817521
45.894231688461
44.0191491987440
43.9607732813441
58.234633242580
37.5407047512380
44.5702757032450
59.2255231721590
48.6759280546490
55.6755334299560
48.2216422723480
56.3112702264560
58.0967492977580
44.9119774101450
56.5492758949570
41.4970612735410
60.3314596345600
54.7269486458550
46.4211565384460
46.6091559145470
42.4881285551420
53.0384671845530
49.8818020686500
55.0049948186550
42.350476532420
40.584274201410
49.9951842256500
66.3006188814660
46.219514741846
37.552546372238
41.632694218442
48.714938556149
56.120521902656
54.456276201454
46.907140484847
48.558091647349
52.309184310452
55.76228558256
47.436602825547
54.869129952655
52.980691533653
44.577874531445
52.217507244552
48.801220044749
44.440427144644
49.273254616149
41.967820278742
35.224494594135
51.520938894852
54.51027972355
59.687864803760
44.561981111545
44.960469393845
50.121410721550
48.319253791148
44.067975421844
51.641640210452
49.249203028749
49.687977378950
58.432107279158
58.581328074859
53.24909251553
44.198274180244
46.858286976947
39.100360835439
56.980853868357
47.819300006148
Sheet3
59.8887915353600
62.4386326913620
56.7375879351570
63.1650597521630
62.9446437111630
57.1744364302570
60.1323633114600
59.1549111655590
59.6527185431600
57.721724867580
63.0190221878630
61.7803813535620
61.7776528694620
61.6489116206620
61.2549412531610
58.7638193488590
57.1493070916570
60.1222451829600
61.4004717741610
58.5832528182590
55.865218807560
58.2806411936580
56.8930023897570
63.8092912291640
58.069506546580
63.0841874832630
60.5399124348611
64.58589056516512
58.2221197575812
60.19730350696016
58.26174644115820
63.46626620746313
59.59150500226011
59.7765416992609
60.0340423867603
57.6960725689583
61.0236226444610
64.6473724069650
61.3082967598610
57.4373486111570
57.4145794113570
60.7304970495610
60.8712572708610
58.8826175468590
61.7597858459620
58.1402197591580
60.2419415068600
61.1905694919610
61.1931297195610
59.6292399373600
62.035030775620
60.5662991498610
63.3443939174630
59.4220661392590
57.314716893570
59.1637696439590
62.6348971005630
60.4973480827600
61.8746868591620
58.65912514590
57.3555986799570
59.407532413959
56.724791344657
63.93909431364
58.918033270559
63.133345671863
56.566912159357
60.250265657160
58.784678609859
62.054380274862
64.590147000365
58.096873241658
60.861007265561
64.420144250664
59.045880940759
61.825633262362
58.606556346259
59.709280018660
57.824943420758
60.273721525460
57.7830066258
59.437523001759
61.128667008761
61.366267952161
58.617067831259
60.020577317660
59.142014530759
57.159411577757
59.859510353460
63.034892870363
60.190384525960
59.791932623360
62.480674083962
59.292842858359
56.950027707557
62.154447429362
60.399593318460
58.082148522458
59.711599230160
57.661570887458
Less Precision
Chart5
00
00
00
00
00
10
00
10
20
20
20
20
50
10
50
51
40
81
81
80
61
23
75
32
21
45
87
25
47
28
26
06
13
14
17
06
14
02
04
05
00
01
00
00
00
00
01
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet4
46.9976784087470
37.2231683185370
52.4425730771522
62.764735402632
61.9835021906622
67.331331037671
28.1641236041282
47.6581875671482
60.9502252599611
39.1329935053393
43.0979583951432
33.0956767255333
31.5308910911324
40.2237050264404
42.2649294604422
28.8206878293293
44.3207512842446
45.9595243126464
51.3485305317513
46.3450704854462
46.7300936987474
46.297594862465
63.4264155349632
49.1471554495491
48.138423507481
44.8679260342454
69.7221197595704
58.656729733593
73.7565473071744
43.4509332888434
66.6145582625670
33.8760231796342
55.3894837038552
59.0219145932593
69.1891558643692
49.154829311490
44.7620494822450
56.7513838076572
46.1867615639460
57.5761136031582
35.5581336305360
41.527624843420
34.7842900635352
46.3712298268460
49.6752080733501
50.2811702871500
46.7728399497470
71.9450157601720
32.575172907733
42.635230228243
24.22419255624
64.476700016564
37.202363627137
43.464200542843
57.577136784758
54.667117536955
58.746087587559
55.957417670356
36.281500241636
38.842614584439
56.939944796657
53.226364242453
40.601622812441
47.590521161348
51.315356712451
55.577976480756
51.387149950551
40.890387380241
68.84845914969
54.871981218555
50.722388904351
58.298411557958
58.620077096659
43.634685324744
40.768083080241
61.111887943161
37.988212524238
34.411078912734
57.11324901257
56.384061634956
72.056883608572
64.437546269664
63.039039004163
51.129603788351
50.019508661450
54.537014319855
49.744852630150
39.453249327239
32.251938490732
58.283313945958
54.442244971954
56.179061529156
52.13473185852
39.73069068640
62.381951819462
46.887868292147
41.600782323142
41.788718034542
45.710072653346
45.466384916745
Sheet1
valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60
300030003000
310031003100
322032003200
332033003300
342034003400
351135003510
362036003600
372137003710
381138003820
393039003920
402040004020
413141004120
424042004250
434043004310
442244004450
453345104551
466146304640
474247604781
4831481204881
4923491804980
5043501705061
5152512205123
5220521305275
531453305332
541454405421
554055005545
564356115687
5736570125725
5847580125847
5947590165928
6003600206026
6123610136106
6225620116213
633563096314
642364036417
650465036506
660366006614
672467006702
680268006804
692169006905
700170007000
710071007101
722172007200
730273007300
741074007400
750075007500
760376007601
770077007700
78178007800
79179007900
80080008000
81381008100
82082008200
83083008300
84184008400
85085008500
86086008600
87087008700
88088008800
89089008900
90090009000
Sheet1
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet5
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet7
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet9
66.5623453338670
56.6444306665570
58.5494923749590
53.991343672540
58.4543364917580
37.6646291139381
58.6645889294590
46.708007782471
61.5657974473621
65.0302332966650
73.1708475237730
53.8070459393541
52.5203610474530
61.4636498236610
57.6856997819582
57.9168933351583
75.7720023708761
60.3768491297602
58.5402155289591
78.3032170753783
53.6926542432543
72.7113708004732
59.8466137185600
59.3558276401594
76.101557776764
63.6263145375640
64.3705881581643
57.6487515596586
66.8332724368677
59.3136498233597
52.6392320049533
49.0246726864493
62.0370066522625
50.18662038505
79.3957930605793
58.7278442859594
44.1236785869443
56.6468521961574
69.5723180493702
56.1702997098561
53.4613583719531
46.3361608732460
63.1611534723631
58.3629095368582
65.7125362218660
53.62626113540
56.5382244227573
62.8593831303630
56.689712134571
62.1222149371621
35.2617441118350
59.3113533492593
62.4685959943620
81.1649421544810
44.7476829018451
44.4136709726440
36.9523128634370
71.7225226859720
83.7436324824840
56.1002458804560
49.9916019785500
65.479932951965
66.589880285967
51.479330674351
68.092911230168
66.602249413867
57.9254880658
44.51849023745
70.127769201170
67.634344001468
56.495148479956
53.09213762953
68.626716407969
63.468278464463
65.562787919266
80.858897187381
62.733690962563
48.93340489349
64.79632262865
47.714613780648
65.655897439366
58.434975623258
62.988713276963
63.57082399264
61.313037500961
57.280610841757
46.526336215347
81.208234102281
75.976957001876
70.488224688770
45.220028975245
48.826925800849
65.326774044165
60.735428784561
59.910119186160
61.612283995262
41.183009367741
57.432701093257
50.620631251751
50.024366525250
Sheet8
51.5621208149520
50.0357272256500
48.9833486325490
48.4957594279480
49.0737342159490
50.261218247500
50.2365482032500
48.8936906448490
45.712278178460
46.7639269016470
50.3006198313500
52.6035195333530
50.6633490557510
50.6333561942510
48.834177859490
49.5841608325501
47.7187962941483
51.8372656996526
48.36266852074812
49.22318238534918
49.60524291965017
45.28394255324522
49.54499344315013
49.8679459168503
51.1556812751514
49.8785233402500
51.9424714992521
46.7312123772470
49.8424800651500
50.9221798791510
48.0860184223480
49.1095228324490
49.9050305632500
46.5371534927470
47.1606666804470
54.3186810217540
48.3907400747480
48.4523674357480
46.8429438013470
50.9023165148510
51.4216675481510
48.3375118973480
51.1437691683510
49.0433093444490
50.6900813787510
48.3697171047480
51.3527733245510
48.9379921317490
51.3600924831510
51.2249029169510
49.4692871041490
48.5710360306490
51.5505247575520
50.9635573406510
50.7099265531510
49.8627299647500
48.9925640875490
46.2330184644460
53.93272785540
52.8293061405530
52.2297672331520
51.54124336452
54.260691639554
49.896144850
49.426372596649
51.850607986952
50.018742412150
50.53055600851
46.675869624647
49.702938566850
51.075773070651
48.649147983149
47.661871020748
53.134391590753
51.775611053752
51.643115865652
51.610105755452
48.979167230649
51.614127995752
47.81106907948
50.006043592350
48.70644842349
51.298269580751
50.509189703751
52.302144821452
50.940426616651
49.125207068649
51.751236595752
51.03287675351
51.270254870251
54.399262252354
48.263751876948
46.357983036446
50.85597548651
48.177845554948
48.779521774849
56.232876330656
50.532140802551
50.341690338250
49.40065663449
Sheet6
65.0056496548650
57.3209696691570
60.8248753147610
58.7707633409590
63.3077071748630
53.2927312329530
62.3775396585620
55.0287224237550
55.9214619594560
58.5760314303590
59.0897845237590
54.2942349662540
68.5711235442690
44.5087586436450
63.3339210859630
57.4822912918571
69.304158084690
49.7818815953501
58.874643552591
58.586890797590
69.8715190688701
64.4599050852643
63.5056245906645
65.9267858887662
59.6263682078601
59.0135643202595
68.5255305748697
68.9733293862695
56.2703191159567
57.0830049278578
51.8423782219526
57.5707351041586
58.5415433912593
60.9120776667614
52.1062362773527
68.2275801105686
58.3764564604584
47.2046100791472
65.7252736951664
56.2178367241565
51.0154292593510
62.336387297621
59.834517439600
75.5964517035760
60.7276730685610
64.807507139650
60.3975469554601
71.2473026093710
51.1998475510
64.382231964640
56.7147141433570
60.4325124792600
69.1942274594690
55.0461642584550
66.9574298323670
51.5543185125520
58.255666469580
60.0562272362600
55.0468190946550
69.6492021839700
62.5066856324630
64.557912234365
67.589892447968
67.524954526268
50.931446518551
65.59492036766
69.76300725670
55.817915987656
56.04949607656
57.632883150858
63.749742063564
69.713294275670
61.498674464561
47.508781487148
58.376456460458
61.405564944461
65.994452294566
61.66261543262
52.568850757453
62.701867742963
67.035341695967
64.696962605565
57.088693817257
51.758299984252
68.459310265668
64.098224053564
58.130306266858
55.354005477655
65.173369572765
58.921791757359
64.330254341864
61.341009010561
64.66460278465
55.584139469456
55.63872279456
57.878528574858
60.015838850260
55.218399817555
51.860850059352
63.863131041764
Sheet2
57.8872290032580
63.997341739640
41.8400590104420
47.380170953470
53.8276266423540
47.9565291142481
51.918901944520
55.9200601754561
49.5412531462502
50.89907644512
49.3804090082492
48.2460894898482
55.9029116528565
62.1690391097621
63.2923014462635
48.255666469485
46.2407628118464
56.4707228375568
49.7504664861508
51.1038764569518
48.2206873028486
47.3194485392472
38.7579906499397
55.7383158492563
52.4437190405522
43.2019409243434
49.2311541064498
37.3942942941372
39.6987173997404
46.9776922626472
48.3450106811482
47.2073965182470
40.1883984421401
51.717539817521
45.894231688461
44.0191491987440
43.9607732813441
58.234633242580
37.5407047512380
44.5702757032450
59.2255231721590
48.6759280546490
55.6755334299560
48.2216422723480
56.3112702264560
58.0967492977580
44.9119774101450
56.5492758949570
41.4970612735410
60.3314596345600
54.7269486458550
46.4211565384460
46.6091559145470
42.4881285551420
53.0384671845530
49.8818020686500
55.0049948186550
42.350476532420
40.584274201410
49.9951842256500
66.3006188814660
46.219514741846
37.552546372238
41.632694218442
48.714938556149
56.120521902656
54.456276201454
46.907140484847
48.558091647349
52.309184310452
55.76228558256
47.436602825547
54.869129952655
52.980691533653
44.577874531445
52.217507244552
48.801220044749
44.440427144644
49.273254616149
41.967820278742
35.224494594135
51.520938894852
54.51027972355
59.687864803760
44.561981111545
44.960469393845
50.121410721550
48.319253791148
44.067975421844
51.641640210452
49.249203028749
49.687977378950
58.432107279158
58.581328074859
53.24909251553
44.198274180244
46.858286976947
39.100360835439
56.980853868357
47.819300006148
Sheet3
59.8887915353600
62.4386326913620
56.7375879351570
63.1650597521630
62.9446437111630
57.1744364302570
60.1323633114600
59.1549111655590
59.6527185431600
57.721724867580
63.0190221878630
61.7803813535620
61.7776528694620
61.6489116206620
61.2549412531610
58.7638193488590
57.1493070916570
60.1222451829600
61.4004717741610
58.5832528182590
55.865218807560
58.2806411936580
56.8930023897570
63.8092912291640
58.069506546580
63.0841874832630
60.5399124348611
64.58589056516512
58.2221197575812
60.19730350696016
58.26174644115820
63.46626620746313
59.59150500226011
59.7765416992609
60.0340423867603
57.6960725689583
61.0236226444610
64.6473724069650
61.3082967598610
57.4373486111570
57.4145794113570
60.7304970495610
60.8712572708610
58.8826175468590
61.7597858459620
58.1402197591580
60.2419415068600
61.1905694919610
61.1931297195610
59.6292399373600
62.035030775620
60.5662991498610
63.3443939174630
59.4220661392590
57.314716893570
59.1637696439590
62.6348971005630
60.4973480827600
61.8746868591620
58.65912514590
57.3555986799570
59.407532413959
56.724791344657
63.93909431364
58.918033270559
63.133345671863
56.566912159357
60.250265657160
58.784678609859
62.054380274862
64.590147000365
58.096873241658
60.861007265561
64.420144250664
59.045880940759
61.825633262362
58.606556346259
59.709280018660
57.824943420758
60.273721525460
57.7830066258
59.437523001759
61.128667008761
61.366267952161
58.617067831259
60.020577317660
59.142014530759
57.159411577757
59.859510353460
63.034892870363
60.190384525960
59.791932623360
62.480674083962
59.292842858359
56.950027707557
62.154447429362
60.399593318460
58.082148522458
59.711599230160
57.661570887458
Can we conclude?
Chart6
00
00
20
20
20
11
20
21
11
30
20
31
40
40
22
33
61
42
31
23
43
52
20
14
14
40
43
36
47
47
03
23
25
35
23
04
03
24
02
21
01
00
21
02
10
00
03
00
177
178
079
380
081
082
183
084
085
086
087
088
089
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet4
46.9976784087470
37.2231683185370
52.4425730771522
62.764735402632
61.9835021906622
67.331331037671
28.1641236041282
47.6581875671482
60.9502252599611
39.1329935053393
43.0979583951432
33.0956767255333
31.5308910911324
40.2237050264404
42.2649294604422
28.8206878293293
44.3207512842446
45.9595243126464
51.3485305317513
46.3450704854462
46.7300936987474
46.297594862465
63.4264155349632
49.1471554495491
48.138423507481
44.8679260342454
69.7221197595704
58.656729733593
73.7565473071744
43.4509332888434
66.6145582625670
33.8760231796342
55.3894837038552
59.0219145932593
69.1891558643692
49.154829311490
44.7620494822450
56.7513838076572
46.1867615639460
57.5761136031582
35.5581336305360
41.527624843420
34.7842900635352
46.3712298268460
49.6752080733501
50.2811702871500
46.7728399497470
71.9450157601720
32.575172907733
42.635230228243
24.22419255624
64.476700016564
37.202363627137
43.464200542843
57.577136784758
54.667117536955
58.746087587559
55.957417670356
36.281500241636
38.842614584439
56.939944796657
53.226364242453
40.601622812441
47.590521161348
51.315356712451
55.577976480756
51.387149950551
40.890387380241
68.84845914969
54.871981218555
50.722388904351
58.298411557958
58.620077096659
43.634685324744
40.768083080241
61.111887943161
37.988212524238
34.411078912734
57.11324901257
56.384061634956
72.056883608572
64.437546269664
63.039039004163
51.129603788351
50.019508661450
54.537014319855
49.744852630150
39.453249327239
32.251938490732
58.283313945958
54.442244971954
56.179061529156
52.13473185852
39.73069068640
62.381951819462
46.887868292147
41.600782323142
41.788718034542
45.710072653346
45.466384916745
Sheet1
valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60
300030003000
310031003100
322032003200
332033003300
342034003400
351135003510
362036003600
372137003710
381138003820
393039003920
402040004020
413141004120
424042004250
434043004310
442244004450
453345104551
466146304640
474247604781
4831481204881
4923491804980
5043501705061
5152512205123
5220521305275
531453305332
541454405421
554055005545
564356115687
5736570125725
5847580125847
5947590165928
6003600206026
6123610136106
6225620116213
633563096314
642364036417
650465036506
660366006614
672467006702
680268006804
692169006905
700170007000
710071007101
722172007200
730273007300
741074007400
750075007500
760376007601
770077007700
78178007800
79179007900
80080008000
81381008100
82082008200
83083008300
84184008400
85085008500
86086008600
87087008700
88088008800
89089008900
90090009000
Sheet1
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet5
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet7
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet9
66.5623453338670
56.6444306665570
58.5494923749590
53.991343672540
58.4543364917580
37.6646291139381
58.6645889294590
46.708007782471
61.5657974473621
65.0302332966650
73.1708475237730
53.8070459393541
52.5203610474530
61.4636498236610
57.6856997819582
57.9168933351583
75.7720023708761
60.3768491297602
58.5402155289591
78.3032170753783
53.6926542432543
72.7113708004732
59.8466137185600
59.3558276401594
76.101557776764
63.6263145375640
64.3705881581643
57.6487515596586
66.8332724368677
59.3136498233597
52.6392320049533
49.0246726864493
62.0370066522625
50.18662038505
79.3957930605793
58.7278442859594
44.1236785869443
56.6468521961574
69.5723180493702
56.1702997098561
53.4613583719531
46.3361608732460
63.1611534723631
58.3629095368582
65.7125362218660
53.62626113540
56.5382244227573
62.8593831303630
56.689712134571
62.1222149371621
35.2617441118350
59.3113533492593
62.4685959943620
81.1649421544810
44.7476829018451
44.4136709726440
36.9523128634370
71.7225226859720
83.7436324824840
56.1002458804560
49.9916019785500
65.479932951965
66.589880285967
51.479330674351
68.092911230168
66.602249413867
57.9254880658
44.51849023745
70.127769201170
67.634344001468
56.495148479956
53.09213762953
68.626716407969
63.468278464463
65.562787919266
80.858897187381
62.733690962563
48.93340489349
64.79632262865
47.714613780648
65.655897439366
58.434975623258
62.988713276963
63.57082399264
61.313037500961
57.280610841757
46.526336215347
81.208234102281
75.976957001876
70.488224688770
45.220028975245
48.826925800849
65.326774044165
60.735428784561
59.910119186160
61.612283995262
41.183009367741
57.432701093257
50.620631251751
50.024366525250
Sheet8
51.5621208149520
50.0357272256500
48.9833486325490
48.4957594279480
49.0737342159490
50.261218247500
50.2365482032500
48.8936906448490
45.712278178460
46.7639269016470
50.3006198313500
52.6035195333530
50.6633490557510
50.6333561942510
48.834177859490
49.5841608325501
47.7187962941483
51.8372656996526
48.36266852074812
49.22318238534918
49.60524291965017
45.28394255324522
49.54499344315013
49.8679459168503
51.1556812751514
49.8785233402500
51.9424714992521
46.7312123772470
49.8424800651500
50.9221798791510
48.0860184223480
49.1095228324490
49.9050305632500
46.5371534927470
47.1606666804470
54.3186810217540
48.3907400747480
48.4523674357480
46.8429438013470
50.9023165148510
51.4216675481510
48.3375118973480
51.1437691683510
49.0433093444490
50.6900813787510
48.3697171047480
51.3527733245510
48.9379921317490
51.3600924831510
51.2249029169510
49.4692871041490
48.5710360306490
51.5505247575520
50.9635573406510
50.7099265531510
49.8627299647500
48.9925640875490
46.2330184644460
53.93272785540
52.8293061405530
52.2297672331520
51.54124336452
54.260691639554
49.896144850
49.426372596649
51.850607986952
50.018742412150
50.53055600851
46.675869624647
49.702938566850
51.075773070651
48.649147983149
47.661871020748
53.134391590753
51.775611053752
51.643115865652
51.610105755452
48.979167230649
51.614127995752
47.81106907948
50.006043592350
48.70644842349
51.298269580751
50.509189703751
52.302144821452
50.940426616651
49.125207068649
51.751236595752
51.03287675351
51.270254870251
54.399262252354
48.263751876948
46.357983036446
50.85597548651
48.177845554948
48.779521774849
56.232876330656
50.532140802551
50.341690338250
49.40065663449
Sheet6
65.0056496548650
57.3209696691570
60.8248753147610
58.7707633409590
63.3077071748630
53.2927312329530
62.3775396585620
55.0287224237550
55.9214619594560
58.5760314303590
59.0897845237590
54.2942349662540
68.5711235442690
44.5087586436450
63.3339210859630
57.4822912918571
69.304158084690
49.7818815953501
58.874643552591
58.586890797590
69.8715190688701
64.4599050852643
63.5056245906645
65.9267858887662
59.6263682078601
59.0135643202595
68.5255305748697
68.9733293862695
56.2703191159567
57.0830049278578
51.8423782219526
57.5707351041586
58.5415433912593
60.9120776667614
52.1062362773527
68.2275801105686
58.3764564604584
47.2046100791472
65.7252736951664
56.2178367241565
51.0154292593510
62.336387297621
59.834517439600
75.5964517035760
60.7276730685610
64.807507139650
60.3975469554601
71.2473026093710
51.1998475510
64.382231964640
56.7147141433570
60.4325124792600
69.1942274594690
55.0461642584550
66.9574298323670
51.5543185125520
58.255666469580
60.0562272362600
55.0468190946550
69.6492021839700
62.5066856324630
64.557912234365
67.589892447968
67.524954526268
50.931446518551
65.59492036766
69.76300725670
55.817915987656
56.04949607656
57.632883150858
63.749742063564
69.713294275670
61.498674464561
47.508781487148
58.376456460458
61.405564944461
65.994452294566
61.66261543262
52.568850757453
62.701867742963
67.035341695967
64.696962605565
57.088693817257
51.758299984252
68.459310265668
64.098224053564
58.130306266858
55.354005477655
65.173369572765
58.921791757359
64.330254341864
61.341009010561
64.66460278465
55.584139469456
55.63872279456
57.878528574858
60.015838850260
55.218399817555
51.860850059352
63.863131041764
Sheet2
57.8872290032580
63.997341739640
41.8400590104420
47.380170953470
53.8276266423540
47.9565291142481
51.918901944520
55.9200601754561
49.5412531462502
50.89907644512
49.3804090082492
48.2460894898482
55.9029116528565
62.1690391097621
63.2923014462635
48.255666469485
46.2407628118464
56.4707228375568
49.7504664861508
51.1038764569518
48.2206873028486
47.3194485392472
38.7579906499397
55.7383158492563
52.4437190405522
43.2019409243434
49.2311541064498
37.3942942941372
39.6987173997404
46.9776922626472
48.3450106811482
47.2073965182470
40.1883984421401
51.717539817521
45.894231688461
44.0191491987440
43.9607732813441
58.234633242580
37.5407047512380
44.5702757032450
59.2255231721590
48.6759280546490
55.6755334299560
48.2216422723480
56.3112702264560
58.0967492977580
44.9119774101450
56.5492758949570
41.4970612735410
60.3314596345600
54.7269486458550
46.4211565384460
46.6091559145470
42.4881285551420
53.0384671845530
49.8818020686500
55.0049948186550
42.350476532420
40.584274201410
49.9951842256500
66.3006188814660
46.219514741846
37.552546372238
41.632694218442
48.714938556149
56.120521902656
54.456276201454
46.907140484847
48.558091647349
52.309184310452
55.76228558256
47.436602825547
54.869129952655
52.980691533653
44.577874531445
52.217507244552
48.801220044749
44.440427144644
49.273254616149
41.967820278742
35.224494594135
51.520938894852
54.51027972355
59.687864803760
44.561981111545
44.960469393845
50.121410721550
48.319253791148
44.067975421844
51.641640210452
49.249203028749
49.687977378950
58.432107279158
58.581328074859
53.24909251553
44.198274180244
46.858286976947
39.100360835439
56.980853868357
47.819300006148
Sheet3
59.8887915353600
62.4386326913620
56.7375879351570
63.1650597521630
62.9446437111630
57.1744364302570
60.1323633114600
59.1549111655590
59.6527185431600
57.721724867580
63.0190221878630
61.7803813535620
61.7776528694620
61.6489116206620
61.2549412531610
58.7638193488590
57.1493070916570
60.1222451829600
61.4004717741610
58.5832528182590
55.865218807560
58.2806411936580
56.8930023897570
63.8092912291640
58.069506546580
63.0841874832630
60.5399124348611
64.58589056516512
58.2221197575812
60.19730350696016
58.26174644115820
63.46626620746313
59.59150500226011
59.7765416992609
60.0340423867603
57.6960725689583
61.0236226444610
64.6473724069650
61.3082967598610
57.4373486111570
57.4145794113570
60.7304970495610
60.8712572708610
58.8826175468590
61.7597858459620
58.1402197591580
60.2419415068600
61.1905694919610
61.1931297195610
59.6292399373600
62.035030775620
60.5662991498610
63.3443939174630
59.4220661392590
57.314716893570
59.1637696439590
62.6348971005630
60.4973480827600
61.8746868591620
58.65912514590
57.3555986799570
59.407532413959
56.724791344657
63.93909431364
58.918033270559
63.133345671863
56.566912159357
60.250265657160
58.784678609859
62.054380274862
64.590147000365
58.096873241658
60.861007265561
64.420144250664
59.045880940759
61.825633262362
58.606556346259
59.709280018660
57.824943420758
60.273721525460
57.7830066258
59.437523001759
61.128667008761
61.366267952161
58.617067831259
60.020577317660
59.142014530759
57.159411577757
59.859510353460
63.034892870363
60.190384525960
59.791932623360
62.480674083962
59.292842858359
56.950027707557
62.154447429362
60.399593318460
58.082148522458
59.711599230160
57.661570887458
Confidence Intervals
The estimated effect size (the difference in the sample averages) is valid only for our sample. Each sample will give a slightly different answer. How do we use our sample to make statements about the overall population? A 95% confidence interval for an effect size tells us that, for 95% of any samples that we could have drawn from the same population, the estimated effect would have fallen into this interval. The Standard error (se) of the estimate in the sample captures both the size of the sample and the variability of the outcome (it is larger with a small sample and with a variable outcome)Rule of thumb: a 95% confidence interval is roughly the effect plus or minus two standard errors.
Hypothesis TestingOften we are interested in testing the hypothesis that the effect size is equal to zero (we want to be able to reject the hypothesis that the program had no effect)We want to test:
Against:
Two Types of Mistakes First type of error : Conclude that there is an effect, when in fact there are no effect.
The level of your test is the probability that you will falsely conclude that the program has an effect, when in fact it does not.So with a level of 5%, you can be 95% confident in the validity of your conclusion that the program had an effect
For policy purpose, you want to be very confident in the answer you give: the level will be set fairly low. Common level of a: 5%, 10%, 1%.
Relation with Confidence Intervals
If zero does not belong to the 95% confidence interval of the effect size we measured, then we can be at least 95% sure that the effect size is not zero.
So the rule of thumb is that if the effect size is more than twice the standard error, you can conclude with more than 95% certainty that the program had an effect
Two Types of MistakesSecond type of error: you fail to reject that the program had no effect, when it fact it does have an effect.
The Power of a test is the probability that I will be able to find a significant effect in my experiment if indeed there truly is an effect (higher power are better since I am more likely to have an effect to report)Power is a planning tool. It tells me how likely it is that I find a significant effect for a given sample sizeOne minus the power is the probability to be disappointed.
Calculating Power
When planning an evaluation, with some preliminary research we can calculate the minimum sample we need to get to: Test a pre-specified hypothesis: program effect was zero or not zeroFor a pre-specified level (e.g. 5%)Given a pre-specified effect size (what you think the program will do)To achieve a given powerA power of 80% tells us that, in 80% of the experiments of this sample size conducted in this population, if there is indeed an effect in the population, we will be able to say in our sample that there is an effect with the level of confidence desired. The larger the sample, the larger the power.
Common Power used: 80%, 90%
Ingredients for a Power Calculation in a Simple Study
Picking an Effect SizeWhat is the smallest effect that should justify the program to be adopted:Cost of this program v the benefits it bringsCost of this program v the alternative use of the moneyIf the effect is smaller than that, it might as well be zero: we are not interested in proving that a very small effect is different from zeroIn contrast, any effect larger than that effect would justify adopting this program: we want to be able to distinguish it from zeroCommon danger: picking effect size that are too optimisticthe sample size may be set too low!
Standardized Effect SizesHow large an effect you can detect with a given sample depends on how variable the outcomes is. Example: If all children have very similar learning level without a program, a very small impact will be easy to detectThe standard deviation captures the variability in the outcome. The more variability, the higher the standard deviation isThe Standardized effect size is the effect size divided by the standard deviation of the outcomed = effect size/St.dev. Common effect sizes:
d=0.20 (small) d =0.40 (medium) d =0.50 (large)
The Design Factors that Influence PowerThe level of randomization Availability of a BaselineAvailability of Control Variables, and Stratification. The type of hypothesis that is being tested.
Level of RandomizationClustered DesignCluster randomized trials are experiments in which social units or clusters rather than individuals are randomly allocated to intervention groupsExamples:
Reason for Adopting Cluster RandomizationNeed to minimize or remove contaminationExample: In the deworming program, schools was chosen as the unit because worms are contagiousBasic Feasibility considerationsExample: The PROGRESA program would not have been politically feasible if some families were introduced and not others. Only natural choiceExample: Any education intervention that affect an entire classroom (e.g. flipcharts, teacher training).
Impact of ClusteringThe outcomes for all the individuals within a unit may be correlated All villagers are exposed to the same weatherAll patients share a common health practitionerAll students share a schoolmasterThe program affect all students at the same time. The member of a village interact with each otherThe sample size needs to be adjusted for this correlationThe more correlation between the outcomes, the more we need to adjust the standard errors
Example of Group Effect Multipliers________________________________Intra-Class Randomized Group Size Correlation 10 50 100 2000.00 1.00 1.00 1.00 1.000.02 1.09 1.41 1.73 2.230.05 1.20 1.86 2.44 3.310.10 1.38 2.43 3.30 4.57 ____________________________________________
ImplicationsIt is extremely important to randomize an adequate number of groups. Often the number of individual within groups matter less than the number of groups
Think that the law of large number applies only when the number of groups that are randomized increase
You CANNOT randomize at the level of the district, with one treated district and one control district!!!!
Availability of a BaselineA baseline has three main uses:Can check whether control and treatment group were the same or different before the treatmentReduce the sample size needed, but requires that you do a survey before starting the intervention: typically the evaluation cost go up and the intervention cost go downCan be used to stratify and form subgroupsTo compute power with a baseline:You need to know the correlation between two subsequent measurement of the outcome (for example: consumption measured in two years). The stronger the correlation, the bigger the gain.Very big gains for very persistent outcomes such as LFP;
Control VariablesIf we have additional relevant variables (e.g. village population, block where the village is located, etc.) we can also control for them
What matters now for power is ,the residual variation after controlling for those variables
If the control variables explain a large part of the variance, the precision will increase and the sample size requirement decreases.
Warning: control variables must only include variables that are not INFLUENCED by the treatment: variables that have been collected BEFORE the intervention.
Stratified SamplesStratification: create BLOCKS by value of the control variables and randomize within each blockStratification ensure that treatment and control groups are balanced in terms of these control variables. This reduces variance for two reasons: it will reduce the variance of the outcome of interest in each strata the correlation of units within clusters.Example: if you stratify by district for an IRS programAgroclimatic and associated epidemiologic factors are controlled for The common district government effect disappears.
The Design Factors that Influence PowerClustered designAvailability of a BaselineAvailability of Control Variables, and Stratification. The type of hypothesis that is being tested.
The Hypothesis that is being TestedAre you interested in the difference between two treatments as well as the difference between treatment and control? Are you interested in the interaction between the treatments?Are you interested in testing whether the effect is different in different subpopulations?Does your design involve only partial compliance? (e.g. encouragement design?)
Power Calculations Using the OD SoftwareChoose Power v. number of clusters in the menu clustered randomized trials
Cluster SizeChoose cluster size
Choose Significance Level,Treatment Effect, and CorrelationPick a : levelNormally you pick 0.05Pick d :Can experiment with 0.20Pick the intra class correlation (rho) You obtain the resulting graph showing power as a function of sample size.
Power and Sample Size
Conclusions: Power Calculation in PracticePower calculations involve some guess work. At times we do not have the right information to conduct it very properlyHowever, it is important to spend effort on them:Avoid launching studies that will have no power at all: waste of time and moneyDevote the appropriate resources to the studies that you decide to conduct (and not too much).