Basic Statistics for Research on Your Teaching

Dr. Herle McGowan, Department of Statistics

October 15, 2010

Basic vocabulary

• A variable is any characteristics you are interested in learning about.

• There are two basic types of variables/data:– Categorical—where the data are words or categories– Quantitative—where the data are numbers

• What you do with data depends on what type of data it is

Basic vocabulary

Other important terms I will use frequently:• Response/Outcome: The variable you are interested in

learning about. In studies of teaching and learning, response variables are often cognitive (e.g. knowledge based) or affective (e.g. based on attitudes, interest, or perceptions).

• Treatment: The thing you are trying; you would like to see if the treatment is related to the response.

The Clicker Example

• Randomized experiment conducted in large, multi-section intro stats class

• Treatment: Frequency of clicker use– High usage: At least 6 questions asked per class – Low usage: 2-4 questions asked per class

• Response variables: – Statistical knowledge, as measured by score on the

Comprehensive Assessment of Outcomes in a first Statistics course (CAOS) exam

– Attitudes towards statistics and clickers, self-reported by students

Getting Started

• All research has to start with an idea. – Ex: Students routinely struggle with a particular concept;

you would like to implement a teaching method that might help them understand it better

– Ex: You would like to try a new activity or new technology that you believe might help engage students

• Anything you are concerned about or interested in trying in your classroom could serve as an idea to be explored through more* formal research

Think about your teaching: What concerns or questions do you have?

Framing your Research Question

• Need to shape ideas into a question (or questions) that can actually be investigated

• Pick one of your ideas from above and write a research question about it

Framing your Research Question

• Example: Does technology help students learn?

Framing your Research Question

• Go back to your research question and try to make it more specific.

Design

• Most important step in determining how valid any observed effects are!

• Creating a well-framed research question will make clear certain aspects of your design, such as– What the treatment is– What the outcome is– What comparison is necessary (if any)

Design

• Today: Focus on designs that involve a comparison between two different groups of students

• Need to consider the best possible comparison group• Should be as similar as possible to students who

receive the treatment

Design

• Key consideration: Avoid having the “stronger” students receive (more of) the treatment

• This provides a plausible alternative explanation for any effect you might see– It is not the treatment that is helping, but rather that

those who would have done well anyway are the ones receiving treatment

Avoid Confounding

• Limit through design– Use random assignment to the extent possible

• Control through analysis– For analytic methods to be successful, you need to

measure possible confounders

What would be some possible measures of student strength?

What are other variables—differences between students, sections, or even instructors—that might also be related to

the outcome(s) of an educational study?

Measurement

• Once you know what you want to measure, you need to think about how you can measure each item

• Some variables are straight forward to measure– E.g. Sex, Year in college– Could be collected via student survey

• Other variables are more difficult to measure– E.g. Learning, Attitude, Teaching style

Measurement

• Ways to measure learning:– Course exam/quiz/activity*– Standardized exam (e.g. Force Concept Inventory, CAOS)

* Assessment of learning doesn’t always have to mean some type of numeric score. You might get a better idea of how much students have learned by using their words instead.– For examples, check out the LITRE project website:http://litre.ncsu.edu/sltoolkit/Main%20SLTK%20Table.html

Measurement

• Ways to measure attitudes or other affective outcomes:– Likert scale• Rate this statement using the scale below:

“I like statistics.”• 1= Strongly Disagree• 2= Disagree• 3= Neutral• 4= Agree• 5= Strongly Agree

Analysis

• First step: Look at your data– Many questions can be answered by a simple graph

• For categorical data, summarize percent of people that fall into each category of the variable, using– Simple table– Bar chart or pie chart

Example

Example

Example

Frequency SD D N A SA

Pre Post Pre Post Pre Post Pre Post Pre Post High <1% <1% 5% 5% 8% 7% 29% 28% 9% 10% Low <1% 1% 4% 5% 7% 6% 29% 27% 8% 11%

Analysis

• For quantitative data, you want to look at three things:1. General pattern of all observations are2. Where most of the data is located (e.g. the center or

typical values—mean or median)3. How spread out the data is (e.g. range or standard

deviation)• Graphs for exploring these features:– Histogram (good for larger data sets)– Dot plot (good for smaller data sets)– Boxplots (good comparing several variables)

Example

Example

Analysis

• If you have two quantitative variables, you can explore the relationship between them with the correlation or by looking at a scatterplot.

Example

Analysis

• Moving on to more formal statistical tests…• Handout provides two tables with examples of statistical

tests that could be used in a variety of scenarios

Regression Analysis

• Allows you to explore relationships between two or more variables– One variable is designated as the response or

dependent variable • What you are interested in learning about• Often denoted Y

– At least one other variable is designated as the predictor or independent variable • What you are using to explain changes in the response• Often denoted X

Regression Analysis

• Can be used with both quantitative and categorical variables– Possible responses of a categorical variable is assigned a

“dummy” code– E.g. 1 if a student falls into that category; 0 if they do not• Referred to as an indicator variable

– Need one less indicator variable than the number of categories

Regression Analysis

• Simplest scenario: One predictor variable; both Y and X are continuous

• Regression model has the form Y = b0 + b1X

– b0 represents the average value of Y when X=0

– b1 represents the effect on Y of a one unit change in X

Example: Exploring the relationship between pretreatment (X) and posttreatment (Y) CAOS scores

• Regression model is estimated to be: Y = 12.5 – 0.662X• The average posttreatment score for those students who received

a zero on the pretreatment test is expected to be 12.5 points.• For every one-point increase in a student’s pretreatment CAOS

score, their posttreatment score is expected to rise by 0.662 points

Coefficientsa

Model

Unstandardized Coefficients

Standardized Coefficients

T Sig. B Std. Error Beta

1 (Constant) 12.500 .539 23.174 .000

score_c1 .662 .025 .623 26.401 .000 a. Dependent Variable: score_c4

Regression Analysis

• Checking how well the model fits the data:1. R-square measures the proportion of the variation in Y

that can be explained by its linear relationship with X. Higher values (closer to 1) indicate better model fit.

Model Summaryb

Model R R Square Adjusted R

Square Std. Error of the Estimate

1 .623a .389 .388 4.058 a. Predictors: (Constant), score_c1 b. Dependent Variable: score_c4

Regression Analysis

• Checking how well the model fits the data:2. Residual plot: Scatter plot of the residuals versus the

independent variable. Good model fit is indicated by a cloud of points centered around zero.

What are residuals?

Residual Plot: Example of Good Fit

Residual Plot

• If you instead see some type of pattern, it indicates a poor fit which might be improved by more sophisticated modeling procedures.

Residual Plot: Examples of Bad Fit

Heteroscedasticity (non-constant variance) Non-linear relationship

Example: Exploring the Effect of Treatment

• Response (Y): Posttreatment CAOS score• Treatment (X): Asking a large (vs. a small) number of clicker

questions– X is categorical; must be turned into indicator variable• X=1 if a student was in the high usage group• X=0 if a student was in the low usage group

• Regression model is Y=b0 +b1X

– b1 represents the effect of being in the treatment group

Example

• Regression model is estimated to be: Y = 26.720 – 0.84X• Students in the low usage group (when X=freq_high=0) are

estimated to have an average CAOS score of 26.72 points• Students in the high usage group are expected to have an

average score that is 0.84 points lower than that

Coefficientsa

Model

Unstandardized Coefficients

Standardized Coefficients

t Sig. B Std. Error Beta

1 (Constant) 26.720 .220 121.193 .000

freq_high -.840 .310 -.080 -2.707 .007 a. Dependent Variable: score_c4

Regression Analysis

• Regression procedure can be modified to deal with any number of scenarios, for example– If more than two groups, more indicator variables can be

included in the model • You will need one less indicator variable than the

number of groups, e.g. 2 indicators for 3 groups– If other variables might confound the relationship

between the response and the treatment variable, they can be included in the regression model• Accounts for effects of confounding variables

Regression Analysis

• In either case, there would be more than one predictor in the regression model– Called multiple regression– Form: Y=b0 +b1X1 +b2X2 + … +bkXk

– b0 represents the average value of Y when each Xi =0

– Each bi , for i=1,…,k, represents the effect on Y of a one unit change in Xi, holding all other variables constant.

Example

• Clearly a relationship between pretreatment and posttreatment CAOS scores

• Would expect that students who had higher beginning knowledge of statistics would also have higher ending knowledge of statistics—even if treatment had no effect

• Thus, it makes sense to include pretreatment CAOS score (X1) in the model when we are trying to estimate the effect of treatment (X2=1 if a student was in the high usage group and X2=0 if a student was in the low usage group)

Example

• Model is estimated to be: Y=12.837+0.660X1-0.556X2

• The average posttreatment score for all students who had a zero on the pretreatment test (X1=0) and were in the low usage group (X2=0) is 12.836 points

• The effect of asking a large number of clicker questions, holding pretreatment score constant, is a decrease of 0.556 points

Coefficientsa

Model

Unstandardized Coefficients

Standardized Coefficients

t Sig. B Std. Error Beta

1 (Constant) 12.837 .559 22.984 .000

score_c1 .660 .025 .621 26.321 .000

freq_high -.556 .245 -.054 -2.269 .023

a. Dependent Variable: score_c4

Tips for Multiple Regression

• Try to keep models small and easy to interpret– Need fewer predictors than the number of people in

your sample– Don’t just “throw” everything into a regression model– Use context and model fit to guide what you include and

what you leave out

Regression Analysis: More Advanced Models

• Each model presented assumed a linear relationship between the response and the predictor variables

• Can account for non-linear relationships by including more complex predictors in the model

• For example– Include polynomial terms, such as X2

– Include interactions between predictors, such as X1*X2

Regression Analysis: More Advanced Models

• Models presented also assumed that, for each value of X, Y was a (continuous) normally distributed variable

• This assumption can be relaxed by using more advanced regression procedures, such as:– Transforming a non-normal response, e.g. log(Y)

Regression Analysis: More Advanced Models

• If Y is categorical:– Using a logistic model or a multinomial logit model for a

nominal response (where the categories have no particular order)• Models the probability of being in a particular category

– Using a cumulative logit model for an ordinal response (where the categories have a natural ordering, such as ratings on a Likert scale)• Models the probability of being in a particular category

i or below

Regression Analysis: More Advanced Models

• Models further assumed that errors in prediction for each observation were uncorrelated

• This may be unrealistic is educational settings, where students often receive treatment as a group– Students nested within a class or under an instructor and thus

share the characteristics of that class/instructor• Account for this with multilevel, mixed effect, hierarchical models• References:

– Raudenbush, S.W. and Bryk, A.S. (2001) Hierarchical Linear Models: Applications and Data Analysis Methods. Thousand Oaks, CA: Sage.

– Pinheiro, J. C. & Bates, D. M. (2000). Mixed-effects models in S and S-PLUS. New York: Springer.

Final Thoughts

• Key is to be principled and *tell others about your design, implementation and findings*

• For more info: – http://www.ncsu.edu/project/fctl/teach-learn/sotl.html – Includes examples, resources, list of journals a possible

publishing outlets

General Regression References

• Chatterjee, S. and Hadi, A.S. (2006) Regression Analysis by Example. Hoboken, NJ: Wiley.

• Harrell, F.E. (2010) Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer.

• Schroeder, L.D., Sjoquist, D.L., and Stephan, P.E. (1986) Understanding Regression Analysis: An Introductory Guide. Newbury Park, CA: Sage.

General Statistics References

• DeVeaux, R.D., Velleman, P.F., Bock, D.E. (2008). Intro Stats. Addison Wesley.

• Rumsey, D.J. (2003) Statistics for Dummies. Hoboken, NJ: Wiley.

• Rumsey, D.J. (2009) Statistics II for Dummies. Hoboken, NJ: Wiley.

• Wikipedia

Software Downloads Through NCSU

• Main page: http://www.ncsu.edu/software/

• SAS or JMP: http://www.ncsu.edu/software/download/sas/

• SPSS: http://www.ncsu.edu/software/agreements/spss.php