1

Notes of Econ424Part 1: Excel

Fall 2007Ginger Z. Jin

http://www.glue.umd.edu/~ginger/

2

Class 1: Introduction

• Goal of this class• Syllabus• First-class questionnaire• Logins• A peek at the data collected by the

questionnaire

3

"There are three kinds of lies – lies, damned lies and statistics."

--- (?) Mark TwainWinston ChurchillBenjamin Disraeli

4

Statistical manipulation– Randomness behind each statistics

• News tend to report outliers or new observation that deviates from the status quo

• Newsworthy is not equal to good statistics

– Correlation and causation• Harvard graduates earn more than high school dropouts, this

does not mean these dropouts, if given a Harvard diploma, will earn as much

– Report favorable information while hide measurement error / variable definition

5

Examples of statistical manipulation• Executives at XYZ Corporation make an average annual salary of

$250,000. – one earns $2 mill, the other 11 earn $90,000.

• 88% of those surveyed prefer QRS brand potato chips.– only surveyed 9 people, 8 said yes.

• ABC ink jet printers use 22% less ink.– compared to what?

• One drug treatment program has a 90% success rate.– Drug free at the end of the program, or x months after finishing the program?

• Graph manipulation: – same data but different scale

Source: http://www.effectivemeetings.com/productivity/communication/statmanipulation.asp

6

Real examples• Kerry: Now, the president has presided over an economy where we've

lost 1.6 million jobs. – there are by 1.6 million private sector job loss since Bush took office, but the drop

in total payroll employment -- including teachers, firemen, policemen and other federal, state and local government employees is down by 821,000.

Source: http://www.factcheck.org/article275.html

• An internet ad by the Democratic Senatorial Campaign Committee: there were "four times as many terror attacks in 2005“ (as compared to 2004).

– National Counter Terrorism Center: The previously used statutory definition of "international terrorism"  ("involving citizens or territory of more than one country") resulted in hundreds of incidents per year; the currently used statutory definition of "terrorism" ("premeditated, politically motivated violence perpetrated against noncombatant targets") results in many thousands of incidents per year.

Source: http://www.factcheck.org/article417.html

7

My goal

At the end of the semester:• You feel comfortable collecting, locating

and analyzing real data• You are able to read, interpret and criticize

statistics generated by other people• Given a real data set, you are able to

generate basic statistics by yourself and give them meaningful interpretations

8

Syllabus is at:http://www.glue.umd.edu/~ginger/Click on “ECON424” at the bottom

Assign project 1

9

Class 2: Introduction to ExcelHandout from UMD peer training• Open a file in N:\ and save it (as .xls or .csv) in

M:\.• Define observation and variable• Hide/unhide rows/columns, freeze panes• Change excel settings (Tools+options)• Change cell/column/row formats• Highlight (shift+, ctrl+) • Formulas/dragdowns• Charts

10

Class 2: More on Excel

Three more items on excel:• Text to columns• Insert text box in excel• Transport excel table and chart into MS

words

11

Class 3: Data Collection (e.stat chapter 3)

• Start with a question– characterize facebook usage among 18-25

• Define observation– an individual, a class, a school?

• Define variables– use or not, intensity of usage, scope of usage

• Define sample– Students enrolled? Students attending the first

class? Students attending the first class and use facebook?

12

Data Collection (continued)

• Methods of data collection– Field collection by hand– Experiment– Survey / Questionnaire

• In class• By phone • By email• Follow up survey

– Existing data sources (library, internet, etc.)

13

Data Collection (continued)

• Things that need special attention– Measurement error

• data collector’s preference, if revealed in the survey, may bias respondent answer

• self report may be biased in a specific direction• anonymous vs. identity-revealing

– Sample selection • sample not representative by design (those attending first

class are different from those who don’t attend)• missing values generate sample selection (need a follow

up?)– Sample size and balance

• variations in the studied variables• trade off between statistical power and cost of data collection• similar size of comparable subsamples

14

Role playing• To better understand subprime mortgage financial

crisis, what statistics would you generate and analyze if you are the head of:

– Federal Reserve – Wall Street Journal– New York Stock Exchange– Countrywide Financial– National Association of Realtors– European Central Bank

15

Background of subprime mortgage financial crisisfinal borrower with poor credit

Bank / Mortgage companies

subprime loans (low introductory rates, insufficient income check, backed up by house value)

depositors investors on corporate debt investors on

mortgage loans

banks issuing loans

……… ………

………

house boom

Fed sets low interest rate

seeking high returns

16

Role playing tasks

• Specify a question• Define a data set that will help you answer the

question. What statistics do you want most from this data set?

• How would you interpret the statistics if it turns out to be …. (imagine the possible outcomes)?

• What cannot the statistics say about? • Identify another statistics that would help you

most but you cannot get internally

17

Class 4: Data Description

Before summarizing the data, clean it first!• Sample – every student that attended 1st class

(may be different from the official roster)• Unit of observation?• # of observations?• Variable(s)?• Missing values?• Abnormal values?

– Delete them, clean them? Be aware of the assumptions you are making.

• What’s the # of observations after all the cleaning?• Take a note of all the above!

18

Class 4: Data Description(e.stat chapter 4)

• Mean

unweighted:

weighted:

Example: compute GPA (e.stat Figure 4.7)

xx

N

ii

N

1

xw x

w

i ii

N

ii

N

1

1

Excel: =average(data)

19

Class 4: Data Description• Median

Define: middle point in the data set50% observations >= median50% observations <= median

Excel: median(data)

If the distribution is symmetric, median=mean.Unlike mean, median is insensitive to outliers.

Example: e.stat Figure 4.8

20

Class 4: Data Description

• Trimmed mean:

Ignores a percentage of values that are extreme and compute mean for the rest.

Excel: trimmean(data, percent)

Example: e.stat Figure 4.10

21

Class 4: Data Description• Order statistics

1st quartile (25% obs below) =quartile(data,1)2nd quartile = median =quartile(data,2)3rd quartile (75% obs below) =quartile(data,3)4th quartile = maximum =quartile(data,4)=max(data)60th percentile (60% obs below) =percentile(data,0.6)0 percentile = minimum =min(data)range = max - minInterquartile range = 3rd quartile – 1st quartileInterquartile ratio = 3rd quartile / 1st quartile

Example: e.stat Figure 4.15

22

Class 4: Data Description• Sample variance and standard deviation

Var(x):

Std dev:

Note: sample variance not equal to population variance

Example: e.stat Figure 4.15

( )x x

N

ii

N

2

1

1

=var(data)

v ar( )x =stdev(data)

23

Class 4: Data Description• Other jargons

Mode: the most common value=mode(data)

Skewness: asymmetry, long right tail = positively skewed

=skew(data)Kurtosis: peakedness, positive if peakier than normal

distribution=kurt(data)

Example: e.stat sections 4-13, 4-18, 4-20.

24

Class 4: Data Description

Exercise: N:\share\notes\datasummary-exercise.xls

Variable: age, gender, # of friends using facebookExercise: use facebook or note, # friends listed

25

Class 5: Histogram (e.stat Section 4-3)

• Histogram: A column chart in which line segments are graphed for the frequencies of classes across the class intervals (bins) and then each segment is connected to the X-axis to form a rectangle.

Frequency of scores

0

2

4

6

8

25 50 75 100

Upper limit of each bin

Coun

t of o

bs in

eac

h bi

n

26

Class 5: Histogram• Steps in drawing histogram:

– Define bins• Must cover all the data (start from min or less, stop at max or more)• Equal width• The number of bins is between 5 and 20

– Count frequency in each bin• Method 1: =countif(..)• Method 2: =frequency(data, bins)• Method 3: Tools – data analysis – histogram

– Plot histogram

• Note: histogram is a frequency chart that shows the distribution of the raw data. It is not equal to highlighting the raw data and plotting them directly.

27

Relative frequency polygon• (Absolute) frequency: number of observations per bin

draw histogram as a column bar chart• Relative frequency: percentage of observations per bin

draw relative frequency polygon as a line chart • Relative frequency polygon is more convenient to compare two data

sets, especially when they differ in the number of observations.

• N:\share\notes\data-summary-example-fall2007.xls

Relative frequency polygons

00.10.20.30.40.50.6

0 25 50 75 100 125

Upper limit of each bin

Rela

tive

freq

28

Class 6-7: Probability Theory(e.stat Chapters 5 and 6)

• Population: entire set of events that occur in a given universe.– Event probability– Probability density function (PDF, )– Cumulative density function (CDF, )– Population statistics (mean, variance, etc.)– Certainty about a random process

• Sample: a subset of a population– Data analysis– Random by nature– Sample statistics are random variables, but population

statistics are not!

f x( )

F x( )

29

Population mean

• For discrete :

• For continuous :

E x prob x xi ieven ts

( ) ( ) x

x

E x x f x dxix

( ) ( )

30

Population variance and standard deviation

• For discrete :

• For continuous :

2 2 prob x x E xieven ts

( ) ( ( ))

x

x 2 2 ( ( )) ( )x E x f x dx

x

31

Compute by hand or excel:

• Bernoulli distribution (flip a coin)

• Flip n coins and define ?• Roll a die

• Roll two dice?• Roll n dice?

E x( ), , 2

x

10

With probability pWith probability 1 p

Each value with probability 1 6/ x 1 2 3 4 5 6, , , , ,

x x x x n 1 2 . . . .

32

How to simulate in excel?Bernoulli Distribution

Hint: =rand() provides a random number between 0 and 1. You can use rand(), but your formula must return integer 0 or 1.

Answer: =if(rand()<p,1,0)

What about flip n coins?

x

10

With probability pWith probability 1 p

33

How to simulate in excel?Roll a die

Hint: Your formula must return an integer between 1 and 6, with equal probability.

Answer: =round(rand()*6+0.5,0), or =randbetween(1,6)Note: randbetween function may not exist in some versions of Excel.

What about roll n dice?

Each value with probability 1 6/ x 1 2 3 4 5 6, , , , ,

34

Compute by hand or excel:

• uniform distribution between (a,b)

f xb a

( ) 1

E x( ), , 2

a b

pdf

x

1/(b-a)

Answer:

E x a b

b a

( )

( )

2

122

2

35

Normal distribution

• Normal distribution:

x N~ ( , ) prob xprob xprob x

( ) .( ) .( ) .

0 6 72 2 0 9 53 3 0 9 9

x N~ ( , )1 0 0 1 0

Normal PDF

0.0000.2500.5000.7501.000

70.0

80.0

90.0

100.0

110.0

120.0

130.0

x

f(x), c

um(p

(x))Example:

36

What about ?

E x x E x E xVar x x Var x Var x Cov x x

x x Var x Var x Cov x x

( ) ( ) ( )( ) ( ) ( ) ( , )

( ) ( ) ( ) ( , )

1 2 1 2

1 2 1 2 1 2

1 2 1 2 1 2

2

2

Note: if both x1 and x2 are normally distributed, so is x1+x2. But if x1 and x2 are uniformly distributed, x1+x2 is not uniformly distributed.

x x x 1 2

37

Normalize

• If uniform on (a,b) then

• If then

x ab a

~ uniform on (0,1)

x N~ ( , )

x N

~ ( , )0 1

x ~

38

How to simulate in excel?• Uniform on (a,b):

• Hint: rand() gives you uniform on (0,1). You need to adjust it to fit in the range of (a,b).

• Answer: =a+rand()*(b-a)

• Normal:

• Answer: =norminv(rand(),miu, sigma)

N ( , )

39

Class 8: Central limit theorem(e.stat section 10-05)

x NN

~ ( , )

Given any population with mean and standard deviation , for a large sample (N>30), we have:

Simulation: show-central-limit-theorem.xls

Assign project 2

Distribution of the Sample Mean

00.050.1

0.150.2

0 50 100X, Xbar

prob

abilit

y

p(x)p(xbar)

40

Show CLT in Excel (1)

• Choose a population– type of distribution, e.g. Bernoulli– distribution parameters, e.g. p=0.3.

• Simulate data – 200 samples– each sample of size N

• Lock in the simulated data so the samples do not change later on – copy, paste special

41

Show CLT in Excel (2)

• Calculate sample mean for each sample• Compare the distributions of (1) all the raw

data and (2) all the sample means– bin range must be wide enough to cover the

most dispersed distribution– bin width must be narrow enough so that

there are at least 4-5 bins for the most concentrated distribution

42

Class 9: Mean estimation

• According to CLT, sample mean is an unbiased estimate of population mean, but with some errors.

• What is the distribution of if ?

Answer:

xE x N( ) , , 0 1 0 10 0

x N E xN

N~ ( ( ), ) ( , ) 0 1

43

distribution of sample mean (xbar)with E(x)=0

0

0.2

0.4

-3 -2.2 -1.4 -0.6 0.2 1 1.8 2.6

xbar

PDF

95% chance within this rangeE x

N( )

2 E xN

( ) 2

44

But …

• Usually we try to guess what is

• We don’t know either• Use the sample to take a guess on

E x( )

prob E xN

x E xN

prob xN

E x xN

( ) ( ) .

( )

2 2 0 9 5

2 2

45

Confidence Interval• Given confidence level (for example)

• Where s=est. std. dev.=stdev(data) t= t value = tinv(1-alpha, N-1)• Or xbar +/- confidence(1-alpha, stddev, size)

• Note: In Excel, confidence function assumes we know population standard deviation and therefore does not use t-value

• Exercise: e.stat problems 12.1 and 12.2

prob x tsN

E x x tsN

( )

0 9 5.

46

Class 10: Hypothesis testing

• Null hypothesis H0:

• Alternative hypothesis: H1:

E x( ) 0

E x( ) 0

E x( ) 0

E x( ) 1

E x( ) 0

two-tail test

one-tail test

47

Logic

• Assume H0 is right• Choose a confidence level • Compute prob(get the sample mean)• Reject H0 if prob(..) is too small, otherwise

accept H0

0 9 5.

48

Types of Error

• Type I Error: reject correct H0 (false neg)• Type 2 Error: accept wrong H0 (false pos)

PDF of Sample Mean: H0 and H1

00.10.20.30.40.5

0.94 0.96 0.98 1 1.02 1.04x

p(x)

p(x)-alt Critical Value p(x)-null mu's

49

In practice• Method 1 (two tail test only)

– Compute and confidence interval– Accept H0 if falls in the confidence interval– Reject H0 if falls out of the confidence interval

x0

0

x

0

0.5

-3 -2.2 -1.4 -0.6 0.2 1 1.8 2.6

PDF

Accept (prob=alpha)Reject

((1-alpha)/2)

Reject

((1-alpha)/2)

50

In practice• Method 2:

– Compute t-statistics

– Compute critical value =(+/-)tinv(1-alpha, N-1) (if two-tail)

=(+/-)tinv((1-alpha)*2, N-1) (if one-tail)

– Accept (reject) H0 if t falls in (out of) the critical value

txsN

0

Degree of freedom

51

0

0.5

-3 -2.2 -1.4 -0.6 0.2 1 1.8 2.6

PDF

Accept (prob=alpha)Reject

((1-alpha)/2)

Reject

((1-alpha)/2)t

Two-tail critical values

52

0

0.5

-3 -2.2 -1.4 -0.6 0.2 1 1.8 2.6

PDF

Accept (prob=alpha)

Reject

(1-alpha)

t

H1:

One-tail critical value

0

53

In practice• Method 3:

– Compute t-statistics

– Compute p-value = prob(t-stat>=|t|)=tdist(|t|, N-1,2) (if two-tail)

=tdist(|t|,N-1,1) (if one-tail)

– Reject H0 if p-value<(1-alpha)

Accept H0 if p-value>(1-alpha)

txsN

0

54

Exercise

• What is H0? What is H1?• Two-tail or one-tail?• Choose a method• E.stat problems:

– two-tail test: e.stat 13.13 and 13.17– One-tail test: e.stat 13.20 and 13.E10

55

Two-tail vs. one-tail

• Two-tail test does not indicate which direction to go if we reject H0: , so the alternative is H1: .

• One-tail test has a strong view of one direction. For example, a saleman wants to know whether sales have increased from the past, in which case the alternative is H1: . If he is worried if the sales have decreased, the alternative will point to another direction where H1 is .

0

0

0

0

56

In Excel (alpha=95%):Two-tail• H0:

• H1:

• T-stat:

• Crit.Val: (+/-) tinv(0.05,N-1)

• Reject if t > + crit. val.

or t< - crit. val

One-tail• H0:

• H1:

• T-stat:

• Crit. Val:= -tinv(0.10,N-1)

• Reject if t<crit. val.

0

One-tail• H0:

• H1:

• T-stat:

• Crit. Val = + tinv(0.10,N-1)

• Reject if t> crit. val.

0

xs N

0

/xs N

0

/

xs N

0

/

0

0

0 0

57

Three methods, same resultTwo-tail• H0:

• H1:

• T-stat:

• Crit.Val: (+/-) tinv(0.05,N-1)

• Reject if t > + crit. val.

or t< - crit. val

0

xs N

0

/

0

Two-tail• H0:

• H1:

• T-stat:

• P-value: tdist(|t-stat|,N-1,2)

• Reject if p<0.05

0

xs N

0

/

0

Two-tail• H0:

• H1:

• Conf. Interval [ , ]

tinv(0.05,N-1)

• Reject if is out of conf. interval

0

0

x ts N /x ts N /

0

58

Class 11: two sample testing

• One sample test H0:• What if we don’t know but have two

samples• Can we compare and ?• Yes, must account for errors in both

0

0

x 1 x 2

59

Two independent samples• H0:• H1: (two-tail) (one-tail)• Independent errors in the two samples are

independent

• Degree of freedom [min(N1,N2)-1]• Exercise: e.stat problems 14.5, 14.6

1 2 1 2

1 2

tx x

sN

sN

( ) ( )1 2 1 2

12

1

22

2

60

Two matched samples(same subjects, N1=N2)

• H0:• H1: (two-tail) (one-tail)• Generate a new variable dx=x1-x2• Transform H0: • Now is a one-sample test• Degree of freedom N1-1• Example: e.stat figure 14.8• Exercise: e.stat problem 14.7

1 2 1 2

1 2

dx 1 2

61

How to tell if two samples are matched or not?

• N1=N2 for matched pairs• Same subjects?• If resorting one sample does not affect the

comparison, they are independent

62

Class 12: regression

• “Regress Y on X” means:

y x Dependent variable

Independent variable(s)

coefficients

Error term

63

Ordinary least squares (OLS)

Scatter Plot: Consumption on Income

0.0010.0020.0030.0040.0050.0060.00

0.00 20.00 40.00 60.00X: Disposable Income

Y: C

onsu

mpt

ion

Goal: Find a linear line that best fits the data

Best: m in ( ) ( ),

y y y xi ii

i ii

2 2

Intercept (random!)

Slope (random!)

Average point is always on the line

( , )x y

( )( )

( )

x x y y

x x

y x

i ii

ii

2

Solution:

64

Test coefficients• Estimated coefficients are random numbers! – depend

on data• The point estimates should be judged together with their

standard errors• Hypothesis test H0: • T-statistics

• Critical values (assuming conf. level=95%)=tinv(0.05,N-2) for two-tail=tinv(0.10,N-2) for one-tail

• P-values =tdist(|t|,N-2, 2 or 1)

0

tstderr

( )

0

Two-tail or one-tail depends on H1

65

Measure the fit of OLS

• Total sum of squared deviations (TSS):

• Decompose TSS:– Explained by the model:– Unexplained residuals:

• R-square: (explained)/(total)

( )y yi

2

( )y yi

2

( )y yi

2

66

F-testk=number of coefficients

N=number of observations

H0: all the coefficients except the constant term are zero.

H1: some of the non-constant coefficients are not zero.

)/()21( )/( dSSUnexplaine --------------------- ---------------------- ),1(

)1/(2 )1/(SExplainedS

kNRkNkNkF

kRk

67

• Correlation coefficient:

( )( )

( ) ( )

x x y y

x x y y

i ii

i iii

2 2

Note:

1. Correlation coefficient is between -1 and 1. What does it mean if correlation coefficient is equal to -1, 0, or 1?

2. correlation coefficient is symmetric, i.e corr(x,y)=corr(y,x), but OLS coefficients aren’t. This means regress y on x is not equivalent to regress x on y.

3. Correlation coefficient is always of the same sign as the OLS slope coefficient.

4. R-square = (correlation coefficient)^2

68

Regression in Excel• Method 1:

=linest(data of y, data of x, include const?, other statistics?)No output labels, must be familiar with the output layout

• Method 2: Tools - data analysis – regression

• Note: you could have multiple x, but they must be adjacent to each other.

• Example: e.stat Figure 19.7• Exercise: e.stat problems 19.E1, 19.E2, 19.E3

69

Assumptions and Caveats in OLS(e.stat section 22-04)

• Errors have mean zero.• Errors have a constant standard deviation.• Errors are drawn independently.• Errors are uncorrelated with x. All the

important x are included in the regression.(omitted variable bias, see e.stat Figure 22.4.)

• Errors are distributed normally.

70

Class 14: Midterm Review

– open book– understand the concepts– use them in real examples– 9:30-10:45am, Plant Sciences 1129– If you cannot attend the midterm for reasons

that are consistent with University Policy, please let me know AT LEAST 12 hours BEFORE the midterm time, otherwise your midterm grade will be counted as zero.

71

Concepts to grasp (1)• Population / sample• Population

– Cdf (prob(var<x))– Pdf (first derivative of cdf)– population mean, population std. dev.

• Sample– Histogram, frequency polygon, quartiles, percentiles, sample

mean, sample std. dev., skewness, kurtosis • Population sample

– Central limit theorem xbar~N(µ, σ/sqrt(n))• Sample Population

– Xbar is a proxy of µ with noise

72

Concepts to grasp (2)

• Inference– Type I error, Type II error– Confidence level α– Confidence interval– Hypothesis testing

• H0• H1• Accept/reject?• One-tail, two-tail test

73

Summary of Excel (1)

• Basic excel – open, save and close files– cut, paste and paste special– change format for cell, row or columns– sort data by one or two variables– chart wizard– freeze panes– drag cells– use excel functions

74

Summary of Excel (2)

• Data description – mean, median, trimmed mean– standard deviation, variance– quartiles– mode, skewness, kurtosis– histogram (absolute frequency)– relative frequency polygon

75

Summary of Excel (3)

• Probability theory – PDF, CDF– mean and standard deviation– bernoulli, binomial– uniform, normal– how to simulate them in Excel?– Central limit theorem– how to see central limit theorem in excel?

76

Summary of Excel (4)

• Estimation and Hypothesis testing– use sample mean to estimate population mean– confidence interval– type I error and type II error– null hypothesis (H0) and alternative hypothesis (H1)– one-tail vs. two-tail– t-statistics, critical value, p-value– one-sample test – two-sample test (independent)– two-sample test (matched pair)

77

Summary of Excel (5)• Linear regression

– model• one variable on the right hand side• more than one variables on the right hand side• create and use binary variables

– fit of the model• R square• F test• scatter plot• correlation coefficient

– coefficient estimates• point estimate• hypothesis testing• omitted variable bias

78

Class 15: Midterm Grades