Survey Methodology – Part IWhy, What, How…And The Evil Twin

Nethra Sambamoorthi, PhDWinter 2014

References for images and concepts are provided.

What Drives Survey Methodology?• It is impossible to do complete enumeration nor it is needed:– Given a time period, specific interests that drive priorities, budget,

and resources, it is impossible to do complete enumeration, neither it is needed; thank goodness science of sampling comes to the rescue, provided

– the behavioral and statistical execution of surveys and statistical adjustments and interpretation of estimates are properly addressed so that it represents the population estimates.

There are Many Ways You Collect Data

• Application/Registration/Enquiry data – Prospect data• Transaction data• Third party – syndicated data (geo-demographic, lifestyle,

attitudinal, behavioral data)• Survey data (special enterprise initiated vs. existing panels)

Ways Bias Comes In our Data Intelligence

– How data were planned to be collected, including training of interviewers – Planning bias

– How questions were designed – Questions as source of bias – Address non-responses and not-full-responses – Design and Data

collection bias– How samples were drawn – Design and data collection bias– How the analytical steps take care of deficiencies from complete

enumeration - design, sample size and power analysis – Analysis bias– Interpretation of estimates and representativeness – Analysis bias

Questions as Source of Bias_1Questions can be a source of bias if they are “bad”:

–Not all respondents understand them the same way.–Respondents do not understand them in the way the researcher intended.–Questions are complex.–One question asks more than 1 question.–Questions include a presupposition. –Question is not applicable to the respondent.–Context changes how question is interpreted.–Questions ask for information that the respondent cannot provide because he doesn’t know it.–Questions ask for information that the respondent cannot provide because it is very likely that he/she forgot it.–Response options do not match the question.

People say something and do something else that may not completely correlate. What matters is what people do. However, what they did may not completely predict the future. It is good to get the preferences and attitudinal aspects of behaviors of your sample respondents so that behaviors assert us with what actually happened. However, that may not be completely predictive for future states, so use the data captured on attitudes and preferences to adjust and create the right narrative for predictive aspects of the future states.REFERENCE: At the end

Design, Sampling, and List As Sources of Bias_2• Reach segments which are likely to provide only rosy picture of

your business hypotheses• Use a list that is not comprehensive or is not representative of

your study population• Reach out more often certain segments compared to other

segments with out adjusting for weights• The randomization of samples are not fully addressed so that

sampling errors are not truly random

Reporting Bias_3

• Reports should comprehensively address the estimates for all the segments of the population

• Provide population estimates which is the goal for sample surveys

Analysis Bias_4• All the right statistical techniques are validated for avoiding

bias in calculations, according to the structure of the questions• Trickiest thing is everything looks good but the sample selected

provides a much a better predictive model. Example in case is using incorrect reference sample.

• Check out the right formulae to use

Where and How Errors Come In…

Survey Methodology – Part IIActual Design and Execution Processes

Nethra Sambamoorthi

Three most Important Aspects of Questions• Behavioral questions (this helps understand what happened in

the past in certain situations)• Preferences (what people like to do in general)• Attitudes (helps understand what drives their behavior for

variations in event differences for future prediction)

Methods of Representativeness• Simple Random Sampling (SRS) and Systematic

sampling• Stratified sampling• Cluster sampling– Multi-stage– Area probability sampling

• Non random sampling – quota sampling– Convenience sampling

Different methods of sample collection requires different methods of analysis and formula

How Many Samples? – Proportion Testing Confidence Intervals, and Margin of Error

• n = required sample sizet = confidence level at 95% (standard value of 1.96)p = estimated prevalence of target event in the project aream = margin of error at 5% (standard value of 0.05)

• n=Sample size = (t2)[p(1-p)]/m2

• sample size needed decreases, if margin of error increases, or confidence level decrease, or the product p(1-p) decreases, which is at the ends of the event likelihood

• For what value of p, we need maximum sample size, other parameters remaining the same. Does it make sense?

• In the formula we have used only one type of error, controlling for false positives(Type I error), but there is also another error, false negatives (Type II error). What would happen if we include that error also in the formula. Will it increase the sample size or decrease? Why marketers do not worry about false negatives? Is it ok, not to be bothered about false negatives in marketing decisions?

Picture Ref: Wikipedia

So how do we take all these different concepts and implement in a survey?

Questionnaire

Survey study design – Plan details

List (sampling frame) and selection

Collect Data and Processing

Design To Implementation and Collecting Data

Define research Objectives

Choose mode of collection Choose sampling frame

Construct and pre-test questionnaire

Fix sampling design and select sample

Recruit and measure sample

Make post survey adjustments

Perform analysis

Define Sampling Plan – Actions to follow

• Identify the parameters to be measured, the range of possible values, and the required accuracy and validity

• Design a sampling scheme that details how and when samples will be taken

• Select sample sizes• Design data storage formats• Assign roles and responsibilities

http://www.slideshare.net/apiong/sampling-design#btnNext

Define Sampling Design – design it for sampling plan

• Define target population• Parameters of interest – summary description of the

population• Sampling frame• Sample size• Appropriate sampling method

http://www.slideshare.net/apiong/sampling-design#btnNext

Survey Methods – Meta Data Define for your survey

Sponsor, broad objective (construct) to achieve Is it a specific department for a specific study or the enterprise?

Third party collaborating with? You may use existing panel or you may commission a special survey for a specific purpose

Purpose (objectives, hypotheses, metrics) Clarity here is extremely important. Clarity here provides validation check point as to whether the rest of the meta data/parameters consistently support this, supporting on all parameters– budget management, bias in expected insights, any thing related to unexpected difficulties in execution, and overall project management.

Survey time period – T The survey unit is defined with in this time period

Survey design – panel/onetime and accordingly associated details

Usually one time for general application. However, survey research companies keep panels that eases on recruitment and management of balanced units across major segments that are needed for intended purposes and also it helps to see trends and dynamics in measurements among the same (similar) groups of people.

Target population Set of finite units(persons) of the population which is studied

Sampling frame The list that defines the selection of targets

Sampling Design Simple random, systematic, stratified random, cluster, multistage

Sample Size For sample size calculation refer http://www.crmportals.com/mail_size_presentation.pdf Also, for immediate calculation of sample sizes, use the following sites http://www.ifad.org/gender/tools/hfs/anthropometry/ant_3.htm A simpler Type I error only based sample size estimation, along with extreme stratification based cases can be seen in this document

Use of interviewer or not Yes/no

Mode of administration Select - CAPI, ACASI, CATI, IVR, T-ACASI, Web (page 139); C=computer, A: audio, P: Personal, S=Self I=Interview(ing), T=Telephone, V=Voice, R=Response

Type of computer assistance Explain the selection here

Reporting/Respondent person Unit of responder

Frequency of survey How often

Interviews per round of survey How many interviews with in a specific survey (for example pre vs. post survey with in a survey results in two rounds

Levels of observations (personal/HH) Head of HH, input person, children, HH, within an organization, the marketing or vendor selection individual…

Weblink Weblink for the web survey

Some Popular Panels• Nielson Panel – 100k members for CPG/retail, Media and Entertainment, and Telecom• NPD Panel – 200K panel of CPG/retail• IRI Panel – CPG/Retail panel of 50K families• Many NIH (National Institute of Health) and Human Resources Surveys available for every one to use for free –

very complex multiyear multistage panels of fairly large size; all of these NIH surveys are covered in your book– National crime victimization(NCVS)– National survey of drug use and health (NSDUH)– Survey of consumers(SOC)– National assessment of educational progress(NAEP)– Behavioral risk factor surveillance system (BRFSS)– Current employment statistics (CES)– The National Human Activity Pattern Survey (NHAPS): a resource for assessing exposure to environmental

pollutants.• BIGresearch – close to 7,500 monthly panel• ComScore (Internet surfing panel) – close to 100K select panel was used; they have 1MM internet surfer panel for

markets to use• Gfk MRI panel (Media Research Inc.,) - Close to 50K respondents

The AnalysisThe Basic and Advanced Opportunities

Marketers use surveys to create enterprise wide applicable strategic insights to: (1) develop segmentation schemes, (2) summarize consumer behaviors and attitudes for the whole US population, and (3) use multiple surveys to draw unified views about their target audience.

However, these insights are not directly addressable and scalable to the whole consumer universe which is very important when applying the power of survey intelligence to the one to one consumer marketing problems marketers routinely face.

Acxiom partnered with Forrester Research, creating addressable and scalable applications of Forrester's Technographics ™ Survey and applied it successfully to a number of industries and applications.

See my presentation at PAW conference:

Incomplete Coverage Error

Estimate of the bias

The bias in the estimate due to non-coverage using simple random sampling. = ); The error in the mean due to under coverage is the product of the coverage rate (U/N) and the difference between the mean of the covered and non-covered cases in the target population.Ex: 0.05(14.3-11.1)=0.16 error in estimate due to under coverage. Compared to other biases, this is perhaps one of the least disconcerting, in general. But recall this was a serious problem in POTUS 2012 Election.

BIAS In Predictive Models – How to measure the goodness of a scoring target population?• In response/non-response or propensity vs. non-propensity comparisons to create scores

that are common in B2C and B2B marketing, the model commonly used is• ; • We want to estimate the parameters (a, b) so that we can use the function for scoring and

ranking– Here the bias is - ), where t subscript refers to the true population value and fr subscript

refers to ‘false reference’ based estimate; the bias is measured here in log-odds scale. – So any variable that increases the differences in the estimated differences in the

difference equation above will increase the bias. For example, for active trader scoring, if you include the whole US population as reference, as if it is going to be targeted (when in fact it is not the right reference as almost 75% of them do not have investment accounts and they are typically less wealthy that the attributes of such people in relation to those characteristics will pop up significantly in the models, meaning increasing false positives and false negatives, the errors. You can calculate variable by variable, how much bias is introduced due to the scoring model and also collectively how much total bias is introduced in the score

Case Study: Harvard Institute of Politics – Youth Survey Panel

• Learn to write like this for executive summary of your survey: http://www.iop.harvard.edu/sites/default/files_new/iop_import/file/spring_poll_12_exec_summ.pdf

• The relative comparison of issues is beautifully captured here: http://www.iop.harvard.edu/sites/default/files_new/iop_import/file/spring_poll_12_issues_matchup_chart.pdf

• Think like the following for supporting analysis of results: and report writing: http://www.iop.harvard.edu/sites/default/files_new/iop_import/file/spring_poll_12_topline.pdf

References

• Ref: Avoiding Bias in Research Interview (http://www.childrenshospital.org/cfapps/research/data_admin/Site2846/Documents/Avoiding Bias in the Research Interview.pdf)

• Note: http://www.gfkmri.com/Products/TheSurveyoftheAmericanConsumer.aspx (Previously called MRI)