The Right Questions and Graphical Displays

© Relay Graduate School of Education. All rights reserved. 1

THE RIGHT QUESTIONS AND GRAPHICAL DISPLAYS


AGENDA OBJECTIVES

Agenda and Objectives

• Descriptive statistics• Dispersion• Aggregate data• The right questions and graphics• Data inference

Compare basic descriptive statistics and identify their limitations

Describe common mistakes associated with analyzing "on average" data

Explain the purpose of the Data Narrative analyses

Evaluate research questions against criteria for quality

2 2


In the last activity, we realized our need for disaggregating the data.

In the Data Narrative, we’ll perform two separate disaggregate analyses,

guided by research questions that we pose of the data.


How Does This Relate To The Data Narrative?

This is in your

Handout




Let’s step back and think about research questions that are being

asked of other data.


Research Questions to be “Consumed”

Did people who switched from Geico to 21st Century Insurance save money on

their car insurance premiums?

This is in your

Handout




their car insurance premiums? Were smokers who used

Chantix more likely to be quit on weeks 9 through 12 than smokers who took the

sugar pill?




their car insurance premiums? Were smokers who used

Chantix more likely to be quit on weeks 9 through 12 than smokers who took the

sugar pill?

What’s the

answer? Let’s find

out!


On The Next Slide, Click “Play” On The Screen

Did people who switched from Geico to 21st Century Insurance save money on their car insurance premiums?


On The Next Slide, Click “Play” On The Screen

Do smokers who use Chantix have a higher probability of being quit on weeks 9 through 12 than users who took the sugar pill?


“There is no such thing as a stupid question…


“There is no such thing as a stupid question…

Actually, yes there is.”


What is Wrong With These Questions?



Were smokers who used Chantix more likely to be

quit on weeks 9 through 12 than smokers who took the

sugar pill?

Click ahead when you’ve completed the appropriate section

of your Handout





SAMPLE BIAS





SAMPLE BIAS

This question is laden with sample bias. It’s only asking about people who switched. Why did people switch from Geico in the first place? Probably because they could save money.

The question didn't say, "for the average person, when shopping for car insurance, was Geico or Century 21 cheaper?"





SAMPLE BIAS

This question is laden with sample bias. It’s only asking about people who switched. Why did people switch from Geico in the first place? Probably because they could save money.

The question didn't say, "for the average person, when shopping for car insurance, was Geico or Century 21 cheaper?"

This is a stupid question. It’s like

asking,

“Do students who attend after-school tutoring do worse than students who never needed to attend tutoring?”





sugar pill?

LEADING QUESTION





sugar pill?

LEADING QUESTION

This is a leading question. They stacked the deck. Why weeks 9 to 12? Why not weeks 6 through 10? More than likely, they retroactively imposed the question when they noticed

that the results were favorable to their position.

The question didn't say, ”in a comparison using a randomized controlled trial, did people who used Chantix quit smoking and stay stopped more than the sugar pill, including people who didn’t quit at all, people who quit and started up again, people who dropped out of the

study, people who quit, started up again, stopped again, and kept going, etc. etc.?"





sugar pill?

LEADING QUESTION

This is a leading question. They stacked the deck. Why weeks 9 to 12? Why not weeks 6 through 10? More than likely, they retroactively imposed the question when they noticed

that the results were favorable to their position.

The question didn't say, ”in a comparison using a randomized controlled trial, did people who used Chantix quit smoking and stay stopped more than the sugar pill, including people who didn’t quit at all, people who quit and started up again, people who dropped out of the

study, people who quit, started up again, stopped again, and kept going, etc. etc.?"

This is a stupid question. It’s like studying your data, noticing some unexpected trend, then

asking a research question about that unexpected trend, already knowing the answer.

The rubric for the Data Narrative does not ask you to disaggregate your data in a way to make you look good. Your research question should

meet its capacity to surprise.


What Makes a Great Research Question?

https://c2.staticflickr.com/8/7116/7453374794_0bab20ec4c_h.jpghttps://upload.wikimedia.org/wikipedia/commons/1/15/Red_Apple.jpg

http://upload.wikimedia.org/wikipedia/commons/3/3a/College_graduate_students.jpg


Minable: Can this Question be Thoroughly Explored?


Did my ELLs in 2nd hour improve on spiraled MC questions from IA #2 to IA #3?



Did my ELLs in 2nd hour improve on spiraled MC questions from IA #2 to IA #3? Not minable.

We don’t want a yes/no question.

Plus it’s too narrow. Only ELLs, only

spiraled MC questions, only IA

#2 vs. IA #3.




How did my students’ performance on spiraled standards vary from assessment to assessment?

Now this is a minable question!

Notice it starts with “How did” not “Did”


Crisp: Is it Succinct and Easy to Understand?



How did boys perform versus girls, overall, across reading and math, relative to question type MC vs OR, and with respect to the different assessments that I administered for my Pathway, including unit and interim?


How did boys perform versus girls, overall, across reading and math, relative to question type MC vs OR, and with respect to the different assessments that I administered for my Pathway, including unit and interim?

Not succinct.

We’re not sure what the question is

asking.




What, if any, were the achievement differences between boys and girls in my class?

Now this is a succinct question!

Notice that the question doesn’t lead…it doesn’t

assume to know the answer already


Meaningful: Is There Rationale for Why I Should Care?


Were the trends in academic performance any different for students who preferred for Ruffles potato chips over Lays in our “Harvest Fair Taste Test”?



Were the trends in academic performance any different for students who preferred for Ruffles potato chips over Lays in our “Harvest Fair Taste Test”?

Not meaningful.

It’s difficult to imagine a rationale for why we should

care about the answer. We don’t

care. We don’t want to know the answer.




How did my students’ performance vary for those in both my class and also in my advisory, versus those who were only in my class?

Now this is a meaningful question! We can envision a solid

rationale for why we should care about

performance of students who spent more or less time with the teacher


Research Question Matchup


Research Question Matchup!

Q1: Did basketball players score higher than actors on standard 6.1.d?

35

Q2:What (if any) differences in achievement existed between athletes and entertainers for math standards mastery?


of your Handout


Q1: Did basketball players score higher than actors on standard 6.1.d?

37




Q2 is minable.

Q1 is not. Q1 is a narrow, yes/no question. NOT minable.

Q2 wins the matchup!

38




Q1: What, if any, variation of achievement existed between people who use a nickname (like “Bill” Clinton, “Steffi” Graf, or “Tony” Blair) and those who didn’t (like “Michael” Bloomberg)?

39

Q2:What, if any, variation of achievement existed between people who had initials early in the alphabet (like “Bill Clinton” or “Andrew Cuomo”) and those who didn’t (like “Serena Wiliams”or “Justin Timberlake”)?



of your Handout


Q1: What, if any, variation of achievement existed between people who use a nickname (like “Bill” Clinton, “Steffi” Graf, or “Tony” Blair) and those who didn’t (like “Michael” Bloomberg)?

41





This is a tough matchup.

Both questions seem meaningless.

HOWEVER, there is some research to suggest that people with names earlier in the alphabet are more successful in life because they are placed first in line more often, over time. So you could write a compelling rationale that makes Q2 meaningful to the reader of your Data Narrative. It better be compelling though.

So Q2 wins the matchup!



Q1:Why did some students meet the Proficient Goal (for math Standards Mastery) while some didn’t?

43

Q2:For my students who did not meet the Proficient Goal (for math Standards Mastery), were there any commonalities in the content that they struggled to master?



of your Handout


Q1:Why did some students meet the Proficient Goal (for math Standards Mastery) while some didn’t?

45





Q1 is an interesting question, but we don’t have the data we need to answer broad “why” questions. This is a question better-suited for earning your PhD than for writing your Data Narrative.

For Q2 you have the opportunity to find commonalities like standards or question types that these students struggled with the most.

So Q2 wins the matchup!



Research Questions: Your Turn!Take a moment to jot down your ideas.

47


of your Handout

49



We’ll also create graphical displays to visualize the findings in response to

the research questions.


“Figures don't lie…


“Figures don't lie…

but liars figure.”


What Could be Improved About this Display?

52

The bar graphs below display students’ overall math Standards Mastery, comparing boys versus girls for Class #1.


of your Handout



54

Titles!



Unify!




Axes Labels!




Axes Scaling!




What is Better About this Display?

58

CS CT RL SG LL S LJ AC MB SW0

10

20

30

40

50

60

70

80

90

100

Overall Standards Mastery, Boys vs. Girls

BoysGirls

Individuals

Scor

e


What is Better About this Display?

59

How else could we display this data? Ideas?


10

20

30

40

50

60

70

80

90

100


BoysGirls

Individuals

Scor

e


What Information did we Lose by Aggregating?

60

Boys Girls0

10

20

30

40

50

60

70

Avg = 41.5

Avg = 62.75

Avg Standards Mastery, Boys vs. Girls

Avg

Scor

e

n = 6n = 4


of your Handout


What Information did we Lose by Aggregating?

62

Boys Girls0

10

20

30

40

50

60

70

Avg = 41.5

Avg = 62.75

Avg Standards Mastery, Boys vs. Girls

Avg

Scor

e

n = 6n = 4

We can’t see the spread

(dispersion) of scores within those buckets


How Can We Make This Even Better?

63

CS CT RL LJ AC MB SG LL S SW0

10

20

30

40

50

60

70

80

90

100


BoysGirls

Individuals

Scor

e



10

20

30

40

50

60

70

80

90

100


BoysGirls

Individuals

Scor

eA Super Rich Graphic: Four Dimensions!(super rich, get it?)

64

70% - Achieve Fl.80% - Amb. Goal



10

20

30

40

50

60

70

80

90

100


BoysGirls

Individuals

Scor

eA Super Rich Graphic: Four Dimensions!

65

70% - Prof. Goal80% - Amb. Goal


Hope you’re impressed.McKayla Maroney.

55% Standards Mastery.

Class #2.