Upload
norma-rhoda-baldwin
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Quantitative Approaches to Dissertation Research
Dr Meredith Rolfe
February 20 & 2, 2013
Agenda
Scientific framework for answering questions Measurement Case Selection Analysis: Quantitative Methods & Experiments
Data Sources
Examples
MG420 Feb 2013 2
Why did the pot of water boil?
• The water reached 100°C• Phase transition and percolation• Someone turned the hob on• Someone was hungry• Time for breakfast• Porridge is a traditional
breakfast food in this country• Owned a pot• Had water supply• The gas bill had been paid• The hob was not broken• Elevation• Pot had a lid• Copper vs. Stainless steel
MG420 Feb 2013 3
Theoretical System
MG420 Feb 2013 4
So, there are 3 (4) types of questions you might ask…
1. Measurement projects:• How do we measure X or Y (e.g., corporate
reputation or innovation)?• How do we identify successful X or Y (e.g.,
policies or practices)?
2. External validity projects:• Does a previous finding generalize to a new
setting? Does an experiment work with different subjects?
• Meta-analysis projects
5MG420 Feb 2013
A. Causal Questions:• Does a change in X produce a change in Y?
• Does cloud computing improve govt. efficiency?• Does mentoring influence managerial career
advancement?
B. Case Analysis questions:• Ex-post: Is X having expected effect on Y?• Ex-ante: Will X have an effect on Y?
6
Third type of question (cont.)
MG420 Feb 2013
Measurement
7MG420 Feb 2013
Science = Measurement?
In physical science the first essential step in the direction of learning any subject is to find
principles of numerical reckoning and practicable methods for measuring some quality connected with it. I often say that when you can measure what you are speaking about, and express it in
numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the
beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science,
whatever the matter may be.Baron William Thomson Kelvin, Popular Lectures and Addresses 1:73 (originally: Lecture to the Institution of Civil Engineers, 3 May 1883)
MG420 Feb 2013 8
Measurement: Another Perspective
9MG420 Feb 2013
Steps in a Measurement Process
1. Identify theoretical concepts
2. Choose one or more indicators/variablesa. Brainstorm possible variables
b. Choose operational rule or definition
c. Level of measurement
d. Measurement technique
3. Assess Measurement Reliability & Validity
MG420 Feb 2013 10
Step 1) Identify theoretical constructs or concepts
• States• Regime type• Democracy?• Economic power• State capacity• Free press
• Individuals• Income• Skills• Work experience• Education or training• Authoritarianism• Political interest• Job success• Compliance
• Corporations• Performance• Financial success• Size• Positional advantage• Leadership• HR policy• Corporate culture• Hierarchy
• Industry• Level of uncertainty• Profitability
MG420 Feb 2013 11
Step 2) Variables: operational rules to turn observations into data
• State: Democracy• fair elections• GNP/capita
• State: Capacity• size of military• Total expenditures
• Individual: Job success• Income• # Promotions• Job title• Compliance
• Behavioural• Attitudes
• Corporation: Performance• Growth• Profits• Ranking• ROI
• Corporation: Positional Adv.• Network Centrality• Categorical
• Corporate Culture/Hierarchy• Work practices • Work policies• Emp/Mgmt attitudes• Network Analysis
MG420 Feb 2013 12
Variables:Levels of measurement
• Nominal • Algeria, Russia (country)• gender or ethnicity of respondent• Revolution, civil war (0/1)
• Ordinal• democracy in 7 categories• war• Corporate rank• Education (degree)
• Interval• agree/disagree 5 point scale
• Ratio (Continuous)• income• length of war• corporate profits
MG420 Feb 2013 13
Discrete “Variables”
SuccessfulUnsuccessful
Net ProfitsMG420 Feb 2013 14
Measurement techniques
• Ordering and counting observed phenomena (words, elections, dollars, number of ads placed, and so on)• Sources: government statistics, corporate statistics,
corporate reports, newspapers, archives• Content analysis (text, verbal)• Categorical measures: Is there a work from home policy?
Are there more than two major parties?• Observed behaviours: How many safety violations are
reported by employees? How many patents are registered by companies? How many polluters are identified and prosecuted by the govt? How many emails are sent by employees? How many times does a person smile (gesture) during a conversation?
MG420 Feb 2013 15
Examples of existing data sources
Opportunity: Old data, new questions
Data Library LSE http://www2.lse.ac.uk/library/eresources/data/Home.aspx
Microdata - Government surveys, longitudinal data and opinion polls
Aggregated Data - OECD, EUROSTAT, IMF, World Bank. Includes Census data
Financial Databases - Information on markets, companies, exchange rates
Geographic Information Resources, GIS - Covering EU admin boundaries and Ordnance Survey data
International Data Centres and Statistical Institutes - Government statistics
MG420 Feb 2013 16
Measurement Techniques 2
• Asking people (perceptions or self-reported behaviours)• Survey questions
• Fixed format• Open-response (requires coding)
• Interviews (elite or random sample)• Sorting tasks (measurement projects)• Implicit attitudes (psychology, often experimental)
• Experimental Manipulation (Independent variable)• Media message (newspaper or video clip)• Assignment to a role• Priming task (think about X)• Information provided or framing
17MG420 Feb 2013
Small survey: Example
• Finding the first job in China (previous dissertation)
• Started with theories about how people find jobs in general, and about the advantages/ disadvantages of using various channels: ads, firms websites, informal channels (strong vs. weak ties)
• Formulated hypotheses
• Applied a questionnaire to 80 young Chinese
• Analysis (in this case it was only frequency tables)
• Discuss to what extent the analysis confirms/ disconfirms the hypotheses and why
MG420 Feb 2013 18
Political Discussion Name Generator
MG420 Feb 2013 19
Self-Monitoring
MG420 Feb 2013 20
Corporate Ranking: Admiration
MG420 Feb 2013 21
Corporate Ranking: Trust
MG420 Feb 2013 22
Measurement techniques 3
• Statistical: Some measurement techniques extract underlying constructs from existing quantitative data• Principle Components• Factor Analysis• Network Analysis
23MG420 Feb 2013
Source: Krebs, orgnet.com
Partisanship & book purchases
MG420 Feb 2013 24
Source: Ronald Inglehart and Christian Welzel, M
odernization, Cultural Change and D
emocracy N
ew York: Cam
bridg University
Press, 2005: page 63.
MG420 Feb 2013 25
Measuring social
meaning &
mental m
aps
MG420 Feb 2013 26
Step 3) Assess Measurement Reliability & Validity
Reliable measures are…
• consistent (repeated measurement gives similar results for the same case)• inter-coder reliability (qualitative data)
• unbiased (right answer, on average)
• precise/low error (close to the right answer)
MG420 Feb 2013 27
Reliable Measures
Consistent
MG420 Feb 2013 28
Step 3) Assess Measurement Reliability & Validity
Reliable measures are…
• consistent (repeated measurement gives similar results for the same case)• inter-coder reliability (qualitative data)
• unbiased (right answer, on average)
• precise/low error (close to the right answer)
MG420 Feb 2013 29
Reliable Measures
ConsistentConsistentUnbiased
MG420 Feb 2013 30
Step 3) Assess Measurement Reliability & Validity
Reliable measures are…
• consistent (repeated measurement gives similar results for the same case)• inter-coder reliability (qualitative data)
• unbiased (right answer, on average)
• precise/low error (close to the right answer)
MG420 Feb 2013 31
Reliable Measures
ConsistentConsistentUnbiased
ConsistentUnbiased
Low VarianceMG420 Feb 2013 32
Valid measures are…
• Content or substance of measure captures construct• face validity – ask an expert
• “what are you thinking about” probes
• manipulation checks (experiments)
• Criterion-related validity• Predictive: Correct relationship with some
predicted outcome variable
• Decision validity: Useful or pragmatic applications
MG420 Feb 2013 33
Valid Measures are… (cont.)
• Construct validity• Convergent validity: Related to other
theoretically related variables/indicators (correlations, alpha scores, factor analysis)
• Example: GNP/capita, tax revenue/capita
• Discriminate validity: empirically discriminates between related constructs (correlations, factor analysis)
• Example: poverty, education
MG420 Feb 2013 34
Measurement: What can go wrong?
Example 1: Do female managers have more empathy than male managers?
== What is empathy? How will it be measured? Behavioural empathy or self-reported expressions of concern? Could these two measures be tapping into different constructs? (Previous studies: YES!)
Example 2: Exposure to ethical discourse will change discriminatory practices in HR.
== Change is what? decrease, increase, both (contingencies)
== What is ethical discourse? What is exposure to ethical discourse (reading an article, participating in a conversation, seeing a poster, overhearing colleagues)
MG420 Feb 2013 35
Questions?
• What constructs might you want to measure?
• How would you would measure them?
36MG420 Feb 2013
Case Selection
37MG420 Feb 2013
Theoretical System
MG420 Feb 2013 38
Universe of Cases orPopulation of Interest
The universe of cases or population of interest includes all of the entities we wish to make generalizations about
• All large (>10k employees) companies• All tech start ups• Employees (in general)• IT professionals• “the public”• Public reform episodes (country-year-policy)
39MG420 Feb 2013
Learning about the population of interest
• We want to learn about the population of interest or universe of cases
• BUT we can’t study all cases in the population (e.g., employees, the public)
• BUT we don’t believe that what we observe represents population of interest (all large companies that have existed or might exist)
• THEREFORE we choose a sample of cases from the larger population or universe of interest
40MG420 Feb 2013
Case Selection Process
1. Define a population of interest
2. Specify a sampling frame
3. Choose a sampling method
4. Choose the sample size* (expert advice)
5. Choose the actual sample of cases
6. Collect data!!
41MG420 Feb 2013
Specify a sampling frame
A sampling frame is a list or enumeration of the population or universe of interest•Phone book•Voter register•Employee records•UN nation list•NYSE or FTSE listing•Professional Assoc. member or licensee
42MG420 Feb 2013
Case or Sample Selection Methods
• Entire universe (e.g., all EU countries, all companies listed on NYSE larger than XX)• Pros: ability to generalize, network analysis• Cons: Time consuming, small-n problem
• Random selection from “universe”• Pros: Best ability to generalize to universe• Cons: Time-consuming, requires data on 20
or more cases (100s-1000s)
43MG420 Feb 2013
Case selection methods
• Convenience samples (LSE students, employees at nearby company)• Pros: Low-cost, easy to gather additional data• Cons: Bias, limited generalizability
• Snowball sample (select initial entities who provide links to additional entities)• Pros: Good when sample frame is not available,
network studies• Cons: Bias, potentially limited generalizability,
hard to identify missing cases
44MG420 Feb 2013
Small-nCase selection methods
These methods are used most often when you study only a few (2-10) cases•Most similar cases (Choose cases with largely similar i.v.s but different outcome)•Least similar cases (Choose cases with largely different i.v.s but same outcome)•Anomalous case* (single case)•Pros: Best for intensive case studies, rare events (e.g., revolutions)•Cons: Sensitive to omitted i.v.s
45MG420 Feb 2013
Sampling: What can go wrong?
• Observations are not independent• Example: using snowball method to test how my
friends and their friends are finding jobs• Selecting on the dependent variable
• Example: Choosing to study three successful companies in order to learn more about how companies become successful (need a few failures too)
MG420 Feb 2013 46
Analysis: Quantitative Methods & Experiments
47MG420 Feb 2013
Theoretical System
MG420 Feb 2013 48
Formulating and testing hypotheses
• Good hypotheses are:• Testable (e.g. identify the dependent and independent variable, and the
relationship between them)• Have a clear null (you test HA against H0)
Example: I believe that employees with artistic training are more innovative.
• Testability: define artistic training, think about the Y (dummy variable; years of training; thresholds); define innovation (e.g. finding a new solution to an old problem)
• Null? Innovativeness is randomly distributed in the population, i.e. the probability to find an innovative employee, p, is not conditionally dependent on an employee’s artistic training
HA: Employees who have more than xxx-years of training in art are more likely to find an innovative solution.
MG420 Feb 2013 49
Lab Experiments: Example
Hypothesis: Competitive environments decrease idea sharing
Construct an experimental design that represents all the elements and conditions specified in the hypotheses
- Select sample of subjects (see previous)- Identify and control non experimental factors (see what literature
says about idea sharing by gender, culture, age etc)- Experimental factors – 2-3 conditions, high/ medium/ low
competition- Validate instruments to measure outcomes (how do you measure
idea sharing)- Design a manipulation check (e.g., ask the subjects how competitive
they believe the situation is)- Pilot study - Run actual experimental trials
Data Analysis: ANOVA
MG420 Feb 2013 50
Survey Experiment on Immigration & Citizenship
Asylum seekers should have the same access to jobs and benefits as everyone else.
Strongly agreeAgreeNeither agree nor disagreeDisagreeStrongly Disagree
Asylum seekers who have been granted citizenship should have the same access to jobs and benefits as everyone else.Strongly agreeAgreeNeither agree nor disagreeDisagreeStrongly Disagree
4%11%
17%
39%
30%
12%30%20%20%18%
MG420 Feb 2013 51
Field EXPERIMENTS
Note: Not all controls factors can be dealt with, but it is expected that large sample size will alleviate some problems
Example: “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination”. M. BERTRAND & S. MULLAINATHAN, American Economic Review, 2004
Hypothesis: There is race discrimination in the US labor market; Whites are favored over African-Americans
Design: - sending fictitious CVs to job ads in Boston and Chicago newspapers- Identical CVs, the only difference is the name (White vs. African-
American)- about 5,000 CVs sentOutcome: White names receive 50 percent more call-backs for
interviews.
MG420 Feb 2013 52
Experiments: What can go wrong?
Example 1: Compare the impact of two treatments (information about product performance and appeals to consumer identity) on consumer intention to buy?
== Ensure that there are no other relevant differences between the two treatments (e.g., use of emotional language) that might affect intention to buy
Example 2: Compare the impact of assignment to organizational role (Manager vs. Employee) on endorsement of ethical principles.
== Include manipulation checks to ensure that the experimental manipulation is working as expected (e.g., through changes in sense of personal power.)
MG420 Feb 2013 53
Observational DataAnalysis methods
• Descriptive statistics (mean, median, sd, tables)• Basic conditional dependence (correlation, Chi-
squared)• Regression (OLS, Probit, Logit, etc)• Time series or event history• Network analysis • Meta-analysis
MG420 Feb 2013 54
Analysis of existing observational data
• Cedric Herring: Does Diversity Pay?: Race, Gender, and the Business Case for Diversity, American Sociological Review, 2009
• Motivation: a wide literature on group diversity, but few studies linking workforce diversity and profitability
• Uses National Organizations Survey, a national sample of for-profit business organizations
• Literature review on the advantages and disadvantages of diverse work force – diversity: (1) gender diversity (2) racial diversity
• 8 hypothesis e.g. Hypothesis: As racial (gender) workforce diversity increases, a business
organization’s sales revenues will increase.
Test: OLS estimation
Sales ($) = f(diversity, controls)
MG420 Feb 2013 55
Event history • When you want to determine how likely is an event, what are the
causes of an event• Example:
• Who is more likely to display unethical behavior, big reputable brands or less known brands (Event = instance of unethical behavior)
• what is the likelihood of unemployment for high skilled professionals (Event = became unemployed)
• Case selection:• Large observation window (min 5+ years --- 10-20)• Relatively large samples (100+)• A combination of firms which had and did not have the event yet
• Censoring issues (you do not observe the entire history of each case)
• Warning! Do not employ event history unless you know that you data satisfies the criteria specified above!
MG420 Feb 2013 56
Meta-Analysis
• When you study a question which has been widely studied, but on which there is little agreement
• “quant literature review”
• Example: Branches (of multinational firms) led by foreign managers perform better/ worse that branches led by nationals.
• What do you do?
• A pure lit review• A meta-analysis
MG420 Feb 2013 57
Meta-Analysis: Steps
- Collect (almost) all relevant quant studies on the topic- If there are too many (hundreds and hundreds) you may want to
sample – e.g. most cited ones, restricted to an area (Asia-Europe)- Snowball samples also work
- Organize all the data in tables - What is the likely direction of the correlation, if any (e.g. is there
any statistically significant evidence, across studies, that manager’s nationality and performance are correlated)
• What is the magnitude of the effect
MG420 Feb 2013 58
What can go wrong?
Despite perfect execution, your hypotheses are not confirmed
What do you do?• Check again that everything is all right with the sample & that the analysis is
correct and complete.
• Do not be afraid to write a paper with “negative results”! Hardly publishable, but of great value to a business!
- revisit the theory to understand nuances- look at the data used in previous studies (what’s the difference between your data and theirs? Old data? Different culture? Technology? etc)- in depth interviews with some of your subjects (see qualitative research class)
• Empirical irregularities allow you to make a contribution to the theory!!!
MG420 Feb 2013 59
Additional ReadingsKilduff, M. & W. Tsai (2003) Social Networks and Organizations. Sage, London
Lewis-Beck, M. S. (1995) Data Analysis: An introduction, Sage (Series: Quantitative Applications in the Social Sciences 103)
Wolf, F. M. (1986) Meta-Analysis, Sage (Series: Quantitative Applications in the Social Sciences 59)
Yamaguchi K. (1991) Event history analysis, Sage (Series: Applied social research Methods 28)
MG420 Feb 2013