Embed Size (px)
DESCRIPTION
After 20 years of conducting research in distributed learning, the presenters address what information data can depict and what they cannot through outcomes demonstrations. The research issues involved in the session include changing baselines, the viral nature of data, complexity, uncertainty, and the notion of blended research.Outcomes: Learn about the evolving issues in contemporary research * Understand that information is best created by combining methods rather than having them compete with each other * Gain insight into blended research http://www.educause.edu/annual-conference/2015/20-years-what-data-show-and-what-they-dont
Citation preview
20 Years: What the Data Show and What They Don’t
Chuck Dziuban & Patsy Moskal
Center for Distributed LearningResearch Initiative for Teaching Effectiveness (RITE)
University of Central Florida
About UCF• Orlando, FL• Over 63,000 students; 2nd largest university in U.S.• Carnegie classification: RU/VH Research University• 216 degree programs; 11 colleges; 11 campuses
• Academic Year 2014-15• 38% of total university student credit hours• 78% of all students took at least one online course
• 80% of all undergraduates (47,116)• 61% of all graduate students (6,469)
The Evaluation PlanStudents
Faculty Institution
Where’s the data??
LMS
ApplicationsSPI
SIS
Surveys
The end goal…
Turning data…
…into INFORMATION
Some Lessons Learned Through the Years
Success
Success
Summer 12 Fall 12 Spring 13 Summer 13 Fall 13 Spring 140
102030405060708090
10090 87 88 91 87 88
94 90 92 94 91 9089 87 90 90 89 89
F2F (n=647,390) Blended (n=73,629)Fully Online (n=189,208)
Perc
ent
1. Does success equal learning?• Answer: No
2. Are the differences significant?• Answer: Yes
Success
Question that was never asked and should have been…
• Is there selection bias in your data?• Answer: Yes
Significance may not be significant
The question for all ages
“How large does my sample have to be?”
Statistical (classical) hypothesis tests are a function of 3 things:
1) Significance Level .05? .01? …or something else?
2) Sample Size
Tiny? Small? Medium? Large? Huge?3) Some Effect Size
A difference that means something to me∆1 –Doesn’t matter
∆2 –Really important to me
How much is enough?
I don’t care about this
∆1
How much is enough?
∆2I care
about this
Statistical significance testing (SD = 15)Sample Size
27502500225020001750150012501000750500
x1=100 x2=101 ES=.06.01.02.03.04.05.07.10.14.20.29
Significant
NotSignificant
So the strategy is…
1) Pick ∆2 first This is important to me
2) Then pick a significance level .05, .01, or something else
3) Pick a sample size that will catch ∆2 but not ∆1
When treatment is not treatment
Blended learning as a boundary object
Blended Learning
Evaluators Journalists
Students
Faculty
DeansLibrarians
Provosts
The way we blendOnline tutorials
E-mails
WEB MODULES
CoMpUtEr LaBs
Optional websites
Tele-web software
Computer cluster room
Internet tutorials
Electronic field trips
Supplemental web
activities
Tech equipped classrooms
Evidence-based practice
Discussion
Virtual experiments
The way we measure
MuLtIpLe ChOiCe
Class ranks
student ratings
Class work
Placement tests
Motivation questionnaires
Achievement tests
Cooperativeness scales
Mathematics anxiety scales
Self-confidence scales
Self-report surveys
APPREHENSION TESTSReading
power
Conceptual testsElectric circuit tests
Can you predict success and
bandwidth analytics?
Add one logistic regression analysis for predicting non-success DFW (n=258,212)
R2
Modality .003Course Level .022Class Size .024Gender .029Ethnicity .035Age .035SAT .034College .047High School GPA .074Cumulative GPA .405
D* 1
Classification and regression tree for predicting non success in online courses
Overall Non Success
D* 3-10
D* 2
13.3%n = 70948
34.3%n = 39646
7.5%n = 31302
24.4%n = 14070
44% n = 25576
* Deciles Prediction Accuracy 94%
D* 1-2
•Can you take action on GPA?•Answer: Yes and No
•Why does GPA predict?
Scarcity and the cognitive bandwidth tax
Could it be scarcity?
Scarcity and the cognitive bandwidth taxC
ogni
tive
Ban
d W
idth
Scarcity Factors
Tuition Housing Books Transportation Child Care
Work Add. Cost
Safety
Scarcity
Scarcity and the cognitive bandwidth taxC
ogni
tive
Ban
d W
idth
Scarcity Factors
Tuition Housing Books Transportation Child Care
Work Add.Cost
Safety
Cognitive Bandwidth
Is there an answer to scarcity?
Harris Rosen
www.tangeloparkprogram.com
Program Components
Early Childhood Education
ParentLeadership
ScholarshipsVocational
Alumni
Tangelo Park Crime Rates 1994-2013Standardized by 1993 Figures
94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 10 11 12 13
Crim
e R
ates
0
20
-20
-40
-60
-80
+3% +4%
-38%-30%
-4%
-33%
-38%-45%
-49%
-67%-61%
-35%-47%
-53%
-45%
-52%
-48%-53%-66%
-66%
Completion rates for those who entered
Community college Vocational school College/ university Graduate school0
102030405060708090
100
32
8377
83
Perc
ent
94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 10 11 12 130
102030405060708090
100
0 0
2836
5867 70 70 68 72 87 76
8288 87 86
76 73 74 74
100
7264
4233 30 30 32 28
1324
1812 13 14
24 27 26 26
Perc
ent
2-yr-old program
Scholarship
TPP Budget: 2-yr-old program and scholarship (percent of total)
Return on Investment to Society
• Total 20 year investment: $10,000,000Half for Early Childhood programHalf for scholarships
• Lance Lochner, University of Western Ontario
• Return on investment: $7 for every $1 spent
Come together blended research
Louis Guttman meets Brené Brown meets the Center for Fiction
Literary Criticism
Image Analysis
Survey
Grounded Theory
Research Initiative for Teaching Effectiveness
For more information contact:Dr. Chuck Dziuban
(407) 823-5478 [email protected]
Dr. Patsy Moskal(407) 823-0283
http://rite.ucf.eduhttp://www.if.ucf.edu/
