Upload
clarence-richards
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
ADULT POPULATION SURVEY (APS) TRAINING SESSIONYana LitovskyGEM Data Team Supervisor
Jonathan Francis CarmonaGEM Data Manager
THIS SESSION WILL COVER:APS tools & guidelines• APS Questionnaire• Survey Design• Weights• Sample Size
Step in APS Cycle:• Proposal Submission• Data Collection• Data Review• Processed Results
the Adult Population Survey (APS) investigates the role of the individual in the lifecycle of the entrepreneurial process and his or her characteristics or actions
a representative sample of the adult population is used to estimate the entrepreneurial activity that represents the entire country.
The APS is unique because it measuresMotives AttitudesActions Business characteristics
SOME APS FACTS
GEM ENTREPRENEURIAL PHASES
APS QUESTIONNAIRE
APS QUESTIONNAIRE• The APS is comprised of MODULES of different groups of questions, often on different topics
• The modules are distributed within the questionnaire in different BLOCKS for different target respondents
• Introduction: All respondents• Block 1: Nascent entrepreneurs• Block 2: Owner-managers• Block 3: Potential and discontinuing• Block 4: Informal investors• Block 5: All respondents (occupation)• Block 6: Reserved (not used in 2014)• Block 7: All respondents (demographics)
APS BLOCKS
APS MODULES Core APS Module
Remain consistent year to year Used to derive main GEM indicators Must be included by all GEM National Teams
Special Topic Module Set of questions that change year to year On different research topics May be included in one or more blocks May include optional questions
Optional Modules Questions on different topics Suggested by national teams or derived from previous Special Topics Follow placement instructions
REQUIRED QUESTIONS FOR NASCENT ENTREPRENEURS
FILTER QUESTIONS
FOR ALL RESPONDENTS
BLOCK 1
QUALIFIED RESPONDENTS
INTRODUCTION
REQUIRED QUESTIONS FOR ENTIRE SAMPLE
OPTIONAL QUESTIONS FOR ENTIRE SAMPLE
BLOCK 1: NASCENT ENTREPRENEURS (FLOW)
UN-QUALIFIED RESPONDENTS
BLOCK 2
OPTIONAL QUESTIONS FOR NASCENT ENTREPRENEURS
SPECIAL TOPIC: REQUIRED QUESTIONS FOR NASCENT ENTREPRENEURS
REQUIRED QUESTIONS FOR OWNER MANAGERS
FILTER QUESTIONS
FOR ALL RESPONDENTS
BLOCK 2
QUALIFIED RESPONDENTS
BLOCK 2: OWNER-MANAGERS (FLOW)
UN-QUALIFIED RESPONDENTS
BLOCK 3
OPTIONAL QUESTIONS FOR OWNER MANAGERS
SPECIAL TOPIC: REQUIRED QUESTIONS FOR OWNER MANAGERS
BLOCK 3
BLOCK 4
BLOCKS 3 & 4
REQUIRED QUESTIONS FOR POTENTIAL & DISCONTINUING
OPTIONAL QUESTIONS FOR POTENTIAL & DISCONTINUING
FILTER QUESTIONS
FOR ALL RESPONDENTS
QUALIFIED RESPONDENTS
UN-QUALIFIED RESPONDENTS
REQUIRED QUESTIONS FOR INFORMAL INVESTORS
OPTIONAL QUESTIONS FOR INFORMAL INVESTORS
BLOCK 5
BLOCK 7
BLOCKS 5 & 7
REQUIRED QUESTIONS FOR ALL (OCCUPATION)
OPTIONAL QUESTIONS FOR ALL (OR BASED ON OCCUPATION STATUS)
REQUIRED QUESTIONS FOR ALL (DEMOGRAPHICS)
REQUIRED VENDOR PROVIDED INFORMATOIN
OPTIONAL QUESTIONS FOR ALL
SPECIAL TOPIC: REQUIRED QUESTIONS FOR ALL (OR BASED ON OCCUPATION STATUS)
• Full questionnaire, with codes and skip patterns.
APS QUESTIONNAIRE FORMAT
APS QUESTIONNAIRE FORMAT• Each block is color-coded
SURVEY DESIGN
• There is no one standard GEM methodology!
• GEM’s only fixed requirement is that a team collects a nationally representative sample of at least 2000 adults
• To ensure this, GEM provides guidelines about:• Proper choice of respondent contact method• Sufficient number of callbacks and initial contact attempts• Appropriate time of survey administration• Appropriate stratification of sample
Some guidelines stable but others differ from team to team
SURVEY DESIGN GUIDELINES
• Sample can be using any combination of:• Face-to-face • Fixed line• Mobile phone
• Choice based on feasibility, budget & bias
• Each methodology can have its own bias, for example:• Fixed-line usually biased toward older respondents• Mobile phone usually biased toward younger respondents • Face-to-face may be biased against people often out of the
house
CONTACT METHOD OPTIONS
• Most teams collect a sample that is proportional to the national population distribution
• Others may add an oversample: may collect disproportionately more responses in a metropolitan area
• Required to meet the n=2000 minimum not counting any oversamples.
• Oversample may be part of Global Report sample or submitted separately
SAMPLE DESIGN OPTIONS
WEIGHTS
• All analysis of GEM APS data MUST use weights.
• National weight mean =1 (do not account for different sample sizes).
• Weights differ by the type and age/range of sample.
• Weighted age/gender distribution must be nationally representative.
• Strata-based weights must match overall strata sizes.
• Very large values for weights (high leverage cases) are not allowed.
WEIGHTS
• RESPONSE WEIGHTS: Used to achieve a better match of the sample surveyed to that of the overall population.• To reduce potential bias due to different responses rates or
normal sampling variability.• Required for every country.
• SAMPLE DESIGN WEIGHTS: Used to adjust a sample that was purposefully designed to have unequal representation• For example, if a team oversamples (by design) in a
metropolitan area, weights are used to not over-represent the city respondents.
TYPES OF WEIGHTS
• WEIGHT_L• Used for the adult population aged 18 to 64 (L = Labor
Force).
• WEIGHT_A• Used for the entire adult population aged 18 to 99 (A
= Adults)
• If national team only collected data for ages 18 to 64, then Weight_L and Weight_A are identical.
GEM NATIONAL APS WEIGHTS
• Only provided for teams who request regional-level analysis.• WEIGHT_LR• WEIGHT_AR
• Each region has its own weights calculated as if they were the entire nation.
• Should be used ONLY for region-to-region comparisons, not for any national-level analysis!
GEM REGIONAL APS WEIGHTS
• Always use weights for any GEM APS individual-level analysis.
• If you want to find comparable Global Report results, use WEIGHT_L
FINAL LESSON
SAMPLE SIZE
SAMPLE SIZES (2013)
COUNTRY SAMPLE SIZE
USA 5698Russia 2029South Africa 3450Greece 2000Netherlands 3005Belgium 2001France 2002Spain 24600Hungary 2000Italy 2052Romania 2021Switzerland 2003United Kingdom 11017
Minimum sampleN = 2000
Appropriate regardless of national population size
RESPONDENTS PER BLOCK (FRACTION OF SAMPLE)
Block 1: Nascent
Entrepreneurs
Block 2: Owner-
Managers
Block 3: Potential and Discontinuing Entrepreneurs
Block 4: Informal Investors
USA 8.8% 10.1% 3.7% 4.4%
Norway 1.7% 11.2% 1.6% 3.2%
Japan 3.0% 7.9% 1.5% 1.2%
Russia 3.7% 6.4% 1.6% 2.0%
South Africa 6.1% 5.7% 4.0% 1.5%
Argentina 10.8% 12.5% 4.9% 3.6%
RESPONDENTS PER BLOCK (NUMBER IN 2000 PERSON SAMPLES)
Block 1: Nascent
Entrepreneurs
Block 2: Owner-
Managers
Block 3: Potential and Discontinuing Entrepreneurs
Block 4: Informal Investors
USA 500 578 209 251
Norway 34 224 32 64
Japan 60 158 30 24
Russia 76 129 33 41
South Africa 210 195 137 52
Argentina 238 276 107 80
• But sample sizes vary by nation and year.
• Beware when combining data from nations or years with different sample sizes.
• Weights do NOT adjust for different sized samples
• The weighted total for any national analysis will match the unweighted total (that is why weights have a mean of 1).
SAMPLE SIZE:COMBINING INDIVIDUAL LEVEL DATA
PROPOSAL SUBMISSION PROCESS
RFP PACKAGE• Request for Proposal (RFP) is posted to the GEM consortium website and an announcement is emailed to all teams in February of each year.
• The RFP includes all documents needed by National Team to submit APS/NES proposal, administer the questionnaires, record and submit all data. It also includes valuable reference information.
“RFP” refers to the entire document package as well as the Request for Proposal document itself.
RFP PACKAGE (CONTENTS)Data Delivery Documents:
SPSS Data Input Templates (APS & NES)
Proposal Documents: Instructions for submitting proposals (APS & NES) Survey Reports (APS & NES) Additional NES instructional documents
Questionnaire Documents: Final version of questionnaires (APS & NES; English & Spanish) APS question-by-question commentary
Reference Documents: PowerPoint Training presentation (APS & NES) GEM Timeline Reference Information (additional APS questions, education categories, optional module)
Use provided SPSS Data Input Template to submit final survey data.
Data in any other format will not be processed
RFP PACKAGEDATA DELIVERY: SPSS DATA INPUT FILE
Detailed description of each question
RFP PACKAGEQUESTIONNAIRE: APS COMMENTARY
~ Excel document comprised of several worksheets
~ Unique for each team
Methodology OverviewPrevious year’s Data Quality ReportAPS Questionnaire informationSample strata description & fieldwork reportPopulation statisticsWeight constructionEducation categories
Additional worksheets populated during APS cycle
RFP PACKAGEPROPOSAL: APS SURVEY REPORT
Worksheet A: Methodology Overview
Provide full details of sample design (usually completed by vendor)
Information included in Global Report
RFP PACKAGEPROPOSAL: APS SURVEY REPORT
- Sample size- Interview method- Sampling methodology- Sample stratification- Call backs- Previous data issues
Worksheet C: Strata & Fieldwork Report
List and describe sample strata (if none used, national information)
Complete fieldwork report for each strata after data is collected
RFP PACKAGEPROPOSAL: APS SURVEY REPORT
PREPARING AN APS PROPOSAL
• Download and carefully read the RFP package
• Work with Survey Vendor to decide on appropriate survey methodology and fill out all required RFP documents.• Vendors must use GEM forms except for Vendor Proposal (has no
template)
• Customize your questionnaire and SPSS Data Input Template• Decide which optional modules you will include (if any)• Decide if you will add additional questions (must be approved by GEM)• Establish you demographic variable answer categories• Translate the questionnaire if necessary
Final questionnaire is each team’s responsibility
TIPS FOR TRANSLATING APS QUESTIONNAIRE
• Work with vendor on translations
• National team has more knowledge of question purpose• Vendor can advise how questions will be received by
respondent• Use question-by-question commentary• Always do a back-translation• Can share other team’s translations with permission
TIPS FOR EDUCATION CATEGORIES
~ Team creates own categories for education variable ~ GEM recodes categories into harmonized variables
Keep our recoding rules in mind when constructing your education variable
- Respondent must complete an education level (e.g. earn a degree) to be coded in that category
- Bachelors, Masters and Doctorate degrees are best kept in separate categories
TIPS FOR INCOME CATEGORIES
- Team creates their own categories for income variable
- GEM requires at least 6 income ranges
- GEM recodes variable into lower, middle, upper percentile
Keep national income distribution in mind! If most respondents fall into one income range, this variable is not useful.
- Email the completed forms to [email protected] (refer to RFP for instructions on how to label documents).
- The GEM Data team will review submitted documents and proposed sampling methodology and send National Team a review in a new worksheet added to Survey Report document (G. Proposal Review).
- The national team MAY NOT BEGIN APS survey until their proposal has been officially approved (via email) by GEM Data Team.
SUBMITTING AN APS PROPOSAL
EVALUATION OF APS PROPOSAL
• Proposal document review: has all the required material been submitted?
• Every mandatory chart applicable to your proposal must be completed• All information must be submitted in required format
• APS Questionnaire review: do team questions interfere with core APS?• Content and placement of any additional APS questions must be approved by Data Team• Verbal introduction of the survey to a potential respondent are reviewed (see RFP tips)
• Methodology review: does sample/survey design meet GEM standards?• Is sample stratification appropriate?• Is the timing of survey administration appropriate?• Is a sufficient number of call-backs and initial contact attempts proposed?• Have all data issues from previous years been addressed?
EVALUATION OF APS PROPOSAL
If necessary, the National Team will be requested to:
submit any missing informationclarify specifics of the sample designrevise sampling and survey plan
DO NOT WAIT FOR PROPOSAL TO BE COMPLETE BEFORE GETTING IN TOUCH WITH
GEM DATA TEAM IF YOU HAVE QUESTIONS
~ We can give suggestions about choice of vendor~ Pre-evaluate different methodologies~ Help team conduct a pilot test
DATA COLLECTION PROCESS
BEFORE APS DATA COLLECTION
• Mentor Team: New teams are recommended to find a find a mentor GEM country. Contact other GEM teams with same cultural background, especially if they have the same language.
BEFORE APS DATA COLLECTION• Pilot Study: New team or teams with new vendor will be
required to conduct pilot study with a minimum of 100 respondents and submit results before continuing data collection.
• Results (submitted using APS Data Input Template) will be tested for:
• proper skip pattern• correct variable coding• proper use of data templates• no excessive missing data
• Any problems will be quickly communicated• If no problems, pilot respondents can be included in final sample. • A team must guarantee that any will be addressed and corrected
DURING APS DATA COLLECTION: HOW TO CHECK YOUR DATA
• Interim Data Submission: If Pilot Study is not required, teams are still encouraged to submit interim data file to be checked for data format, skip logic or other errors.
• Data Quality Syntax: A national team can use SPSS syntax to check their data before submitting to GEM. These files replicate most GEM data checks.
DURING APS DATA COLLECTION: ERRORS TO CHECK AND AVOID
① Skip logic errors due to incorrect questionnaire programing or poorly trained interviewers: If these result in excessive missing data, team may be asked to recontact participants (if possible) or resample, in order for the data to be approved and included in the Global Report (extra data is also a problem)
DURING APS DATA COLLECTION: ERRORS TO CHECK AND AVOID
② Unrepresentative sample due to incorrectly implemented sampling procedure: If final sample does not reflect the age/gender distribution of country within an acceptable range, team’s data may not be acceptable.
RECORDING APS DATASome basic guidelines:
• All recorded data must correspond to a designated answer category
• Each respondent ID is unique and no greater than 9 digits• Do not delete variables from SPSS template• Do not use any digits or symbols to indicate a blank cell
(e.g. “0”)• Add value labels for demographics with team-created
categories
... see all guidelines in RFP and Data Quality Syntax
OPEN-ENDED TEXT RESPONSES
What kind of business is this?
• Response must describe:• The nature of the product or service as well as• The primary customer base: other businesses or typical
consumers• The detailed sector of activity
• Common businesses need little elaboration:• Restaurant, homebuilder, day care center, hair salon, etc.
COMMON PROBLEMATIC RESPONSES
Unacceptable Response Problem and Solution
Apparel Specific sector needed. Can be apparel manufacturing, clothing store, fashion design.
Electrical Appliances Specific sector needed. Can be manufacture electrical appliances, wholesale electrical appliances, retail electrical appliances, repair electrical appliances, etc.
Machines Must probe to know type of machinery and specific sector such as manufacturing, wholesale, retail, repair or rental.
Services Must probe for more information. Services can be just about anything.
Telephones Specific sector needed. Can be selling mobile phones, repairing phones or working for a call center.
Trade Probe for more specific information. Can be anything; construction (building, plumbing, electrician), manufacturing, wholesale or retail.
VERIFY SURVEY REPORT INFORMATIONConfirm that all Survey Report information submitted before data collection began is still correct:
Education categories are the identical in Survey Report and dataset Population data used to calculate weights same as provided in Survey Report Strata listed in Survey Report have the same names, value labels order Correct weight formula was indicated
If any changes must be made to Survey Report, make the necessary revisions and alert GEM Data Team when revised document is submitted.
SUBMITTING APS DATASET- Submit APS Data in SPSS template (follow document naming
conventions indicated in RFP)
- Complete Strata & Fieldwork Report in Survey Report and submit with APS Data
- Number of answered calls/doors - Number of eligible households contacted- Number of interviews agreed to...
- Send both documents to [email protected]
DATA REVIEW PROCESS
INITIAL REVIEW OF APS DATA
Initial data checks
1. Data is correctly recorded and formatted
2. Unique, numeric IDs value for each respondent
3. Proper variable and value labels are used
4. All mandatory questions have been asked
Weights
1. Representative of age/gender distribution in country and, if sample is stratified, representative of strata distribution.
2. No excessive number of high value weights
Open-ended responses
1. Translated into English
2. Both native and English language responses provided in appropriate variables.
Skip logic checks
1. No excessive missing data
2. If data missing, how critical are the errors?
Example of APS Data Review Issues (worksheet H. Data Review, in Survey Report)
DATA REVIEW (SENT TO NATIONAL TEAM)
• Comprehensive review of data quality and team performance
• Included in new worksheet in Survey Report (I. Data Quality Report)
• Includes:• % missing for key variables• % refused for key variables• % skip logic errors• Key indicators for past 3 years• Sample distribution• Deadlines• Exchange rate• Proposal/APS Issues
DATA QUALITY REPORT
DATA QUALITY REPORT
APS PROCESSED RESULTS
APS DATA PROCESSING LEVELS
Submitted Raw Data
HarmonizedData
Processed Individual Level Data
Processed National Level Data
Data submitted by National Team
Reviewed, cleaned, standardized data ready to be merged with all other countries for processing
APS Data with all computed APS indices (TEA rates, Established Business rates, etc.)
APS index rates computed at the national level and weighted by weight_L (for 18-64 population)
COMPUTING APS MEASURES
• Data Team uses cleaned, harmonized and merged datasets to compute indices from core APS data.
• Some individual level data variable names and variables differ from merged national level data.
• Variable codebooks for both individual and national level data made available with processed APS results.
• National level results (weighted by WEIGHT_L) are reported in the Global Report.
TEA CREATION FLOWCHART
During & 1 year after data Collection
1 year after Global Report Launch
Processed individual level data each country Each GEM National Team
All GEM Teams
Processed individual level data all countries
GEM Data, Special Topic Teams,Unilateral Sharing
WHAT DATA IS AVAILABLE?
FebruaryRFP released
mid-MarchNES RFP due
late-AprilAPS RFP (Survey Report and Survey Vendor Proposal) due
mid-JulyAPS & NES data and documentation (including Fieldwork Report) due
mid to late-SeptemberFirst release of APS & NES master dataset on website
mid to late-OctoberSecond release of APS master dataset uploaded to website
TIMELINE
TIMELINESSSUBMIT ALL REQUIRED
MATERIALS ON TIME
DATA QUALITYENSURE THAT YOU FOLLOW ALL
GEM STANDARDS AND PROCEDURES
COMMUNICATIONSTAY IN TOUCH WITH DATA TEAM AT [email protected]