SAMPLING

Next week

2 book chapters Outline of thesis proposal/paper intro Find a scale and answer questions Thought paper

General Sampling Issues

What is the sampling model? What types of biases can come in at

each point? What is the proximity similarity model?

What are issues with that model? How can you increase external validity? When do you need a representative

sample?

Sampling Distributions

How are sampling distributions relevant to research?

What is the difference between the variance, standard deviation, and standard error?

How does the standard error relate to n? SD?

What do 68, 95, and 99 refer to? What are confidence intervals? What do

they mean?

Probability Sampling

Are random samples really random? When would you use each?

Simple random sampling Stratified random Systematic random Cluster Multistage

Nonprobability Sampling

What are these? When should they be used?

Convenience sample Modal instance Expert Quota Heterogeneity Snowball

Census

How do censuses do sampling? What are problems with the national

census? Ways to deal with them?

Power

What is power, and why does it matter? Why do studies get published even if

they are underpowered? What group-level consequence (for science) does this have?

How do people determine power? How should they?

Power vs. accuracy

How does sample size planning for power vs. accuracy differ?

When would you want to do one vs. the other?

Figure 1. For AIPE, effect size doesn’t matter

Power

What stats should we report in a study? What does APA manual say?

How can simulations be used to estimate power? How does power relate to meta-analyses? How does power differ for omnibus vs. specific

tests? What’s post hoc power? What’s the problem with

it? For your papers in here, estimate power and

justify your sample size (don’t just plug into g-power)

Other stat issues

The problem of p http://

www.youtube.com/watch?v=ez4DgdurRPg

The relationship between p, effect size, and n

Practical vs. statistical significance How much power should we have? Registries, multi-site trials Standardized vs. unstandardized effect

sizes

How can you increase power? Increase sample size or alpha Decrease mean square error by using better measures, increasing

control, and getting high quality data Use within-participant designs or use covariates Increase the variance of the IV (use a more powerful treatment) Use orthogonal contrasts or get predictors that aren’t correlated to

each other Ensure that you’re not violating assumptions of your stats Look at theory and previous research to find the best, most powerful

predictors Use a more homogeneous sample Do field studies Increase sample size Treat missing data in a more appropriate way McClelland, 2000; Funder et al., 2014

Funder et al., 2014

SPSP Task Force on Publication and Research Practices

Recommendations: Describe choice of N and issues of power Report effect sizes and 95% CI Avoid “questionable research practices” Give all IV and DV instructions and

measures in an Appendix or online Provide data and coding to those who ask

Get better outlets for replication studies Be open to differences in methods, groups,

etc. Recommendations for education:

Encourage “getting it right” over “finding significant results” (p. 9)

Tell the “whole story” rather than a “good story” (p. 9)

Teach things like power, effect sizes, CI, questionable practices, etc.

Be good models

Cross-cultural research

What is a culture? Why study things across cultures? Best practices:

Have at least one insider on the research team

Match samples on typicality and as many things as you can besides culture

Translation and back translation Are they cultural differences or

miscommunications or differences of response styles?

Henrich, Heine, & Norenzayan, 2010

Sears, 1988 What are WEIRD samples? How much can we generalize our

results? When does generalization make sense?

Why do we focus so much on WEIRD samples?

What should we report about demographics?

Industrialized societies vs. small-scale societies

What kinds of differences exist? Similarities? Why?

Western vs. non-Western

What are differences? Similarities? Why?

Americans vs. other Westerners How are we weird? Why are we weird? What does this suggest about what we

“know” in psychology?

American participants vs. others Differences? How do college students differ from

others? For what topics are less likely or not to

have an effect? How do we differ from Americans in the

past?

Overall

What problems do WEIRD samples cause?

When is it okay to use WEIRD samples? How can we deal with these issues? So is your research just a worthless pile

of ….?