Upload
alex-rayon-jerez
View
334
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Data Analytics process in Learning and Academic Analytics projects. Day 3: Data processing
Citation preview
Data Analytics process in Learning and Academic
Analytics projects
Day 3: Data processing
Alex Rayón [email protected]
DeustoTech Learning – Deusto Institute of Technology – University of DeustoAvda. Universidades 24, 48007 Bilbao, Spain
www.deusto.es
Table of contents
● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery
Table of contents
● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery
Data dimensionsSummary
[Verbert2011]
Data dimensions1) Computing
● Software○ Example
■ Q1. Among the tools, which is more representative of the final grade?
■ Q5. Which is the impact of the social networks in the group composition?
■ Q6. Which tools are more prone to foster collaboration?
■ Q7. The use of some collaboration tools has effect on the final grade?
● Hardware● Network
Data dimensions2) Location
● Quantitative● Qualitative
Data dimensions3) Time
● Timestamp● Time interval
Data dimensions4) Activity
● Events● Tasks● Goals● Subject
○ Example
■ Q2. Which are the differences in terms of grades
between this subject and other subjects where we already know the final grade?
Data dimensions5) Physical condition
● Noise level● Lighting● ...
Data dimensions6) Resource
● Physical resource● Virtual resource
Data dimensions7) User
● Basic info○ Example
■ Q3. Is there any gender difference in the use of the tools?
● Knowledge● Interest● Goals
○ Short-term○ Long-term
● Learning styles● Affects● Background
Data dimensions8) Relations
● Social relations○ Example
■ Q4. Are there groups of people that repeatedly collaborate in different tools?
■ Q4. Do these groups repeat over time?
● Functional relations● Compositional relations● Proximity● Orientation● Communication
Table of contents
● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery
ApplicationsWhy do learners use analytics?
[Ferguson2014]
● Monitor their own activities and interactions● Monitor the learning process● Compare their activity with that of others● Increase awareness, reflect and self reflect● Improve discussion participation● Improve learning behaviour● Improve performance● Become better learners● Learn!
ApplicationsWhy do teachers use analytics?
[Ferguson2014]
● Monitor the learning process● Explore student data● Identify problems● Discover patterns● Find early indicators for success● Find early indicators for poor marks or drop-
out● Assess usefulness of learning materials
ApplicationsWhy do teachers use analytics? (Ii)
● Increase awareness, reflect and self reflect● Increase understanding of learning
environments● Intervene, advise and assist● Improve teaching, resources and the
environment
Table of contents
● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery
Data processingTransform menu
Data processingScripting menu
Data processingJoins menu
Data processingStatistics menu
Data processingWEKA plugin
Data processingWEKA plugin (II)
Data processingWEKA plugin (III)
Data processingWEKA plugin (IV)
Data processingWEKA plugin (V)
Table of contents
● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery
Knowledge discoveryIntroduction
[BakerSiemens2014]
This review draws on past reviews (cf. Baker & Yacef, 2009; Romero & Ventura, 2010; Ferguson, 2012; Siemens & Baker, 2012)
Knowledge discoveryIntroduction (II)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)
Knowledge discoveryClassification
1. Prediction methods
2. Structure discovery
3. Relationship mining
Knowledge discovery1) Prediction methods
● The goal is to develop a model which can infer a single aspect of the data ○ The predicted variable
○ Similar to dependent variables in traditional statistical analysis
● … from some combination of other aspects of the data○ Predictor variables
○ Similar to independent variables in traditional statistical analysis
Knowledge discovery1) Prediction methods (II)
● Prediction models are commonly used: ○ Predict future events (Dekker2009; Feng2009;
MingMing2012)
○ Predict variables that are not feasible to directly collect in real-time
■ Example: collecting data on affect or engagement in
real-time often requires expensive observations or disruptive self-report measures
■ Whereas a prediction model based on student log
data can be completely non-intrusive (Sabourin2011)
Knowledge discovery1) Prediction methods (III)
Source: http://etec.ctlt.ubc.ca/510wiki/Learning_Analytics
Knowledge discovery1) Prediction methods (IV)
● Three types of prediction models are common in EDM/LA:○ Classifiers○ Regressors○ Latent knowledge estimation
Knowledge discovery1) Prediction methods (V)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)
Knowledge discovery1) Prediction methods (VI)
● Classifiers○ The predicted variable can be either a binary (e.g. 0 or
1) or a categorical variable
○ Some popular classification methods in educational domains include:
■ Decision trees
■ Random forest
■ Decision rules
■ Step regression
■ Logistic regression
Knowledge discovery1) Prediction methods (VII)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)
Knowledge discovery1) Prediction methods (VIII)
● Regressors○ The predicted variable is a continuous variable
■ For example: if the Grade can be explained by the number of pending subjects and the call number
○ The most popular regressor in EDM is linear regression
■ Note that linear regression is not used the same way in EDM/LA as in traditional statistics, despite the identical name
Knowledge discovery1) Prediction methods (IX)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)
Knowledge discovery1) Prediction methods (X)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)
Knowledge discovery1) Prediction methods (XI)
● Latent Knowledge Estimation○ Actually is a special type of classifier
○ A student’s knowledge of specific skills and concepts is
assessed by their patterns of correctness on those skills
○ A wide range of algorithms exist for latent knowledge estimation, being the two most popular:
■ Bayesian Knowledge Tracing (Corbett & Anderson, 1995)
■ Performance Factors Analysis (Pavlik2009)
Knowledge discovery1) Prediction methods (XII)
● Classifiers in WEKA are models for predicting nominal or numeric quantities
● Implemented learning schemes include:○ Decision trees and lists, instance-based classifiers,
support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, etc.
● “Meta”-classifiers include:○ Bagging, boosting, stacking, error-correcting output
codes, locally weighted learning, etc.
Knowledge discovery1) Prediction methods (XIII)
Knowledge discovery2) Structure discovery
● Attempt to find structure in the data without an a priori idea of what should be found
● It is, actually, a very different goal than in prediction○ In prediction, there is a specific variable that the
EDM/LA researcher attempts to model;
○ By contrast, there is not a specific variable of interest in structure discovery
○ Instead, the researcher attempts to determine what structure emerges naturally from the data
Knowledge discovery2) Structure discovery (II)
● Include:○ Clustering○ Factor analysis○ Social Network Analysis○ Domain Structure Discovery
Knowledge discovery2) Structure discovery (III)
● Clustering○ The goal is to find data points that naturally group
together, splitting the full data set into a set of clusters
○ Clustering is particularly useful in cases where the
most common categories within the data set are not known in advance
○ If a set of clusters is well-selected, each data point in a
cluster will generally be more similar to the other data points in that cluster than data points in other clusters
Knowledge discovery2) Structure discovery (IV)
● Clustering○ Clusters have been used to group students (Beal2006)
and student actions (Amershi2009)
■ Amershi & Conati (2009) found characteristic
patterns in how students use exploratory learning
environments, and used this information to identify more and less effective student strategies
Knowledge discovery2) Structure discovery (IV)
● Factor analysis○ A closely related method
○ Here, the goal is to find variables that naturally group
together, splitting the set of variables (as opposed to
the data points) into a set of latent (not directly observable) factors
○ Factor analysis is frequently used in psychometrics for validating or determining scales
Knowledge discovery2) Structure discovery (V)
● Factor analysis○ In EDM/LA, factor analysis is used for dimensionality
reduction (e.g., reducing the number of variables) for a wide variety of applications
○ For instance, [Baker2009] used factor analysis to
determine which design choices are made in common by the designers of intelligent tutoring systems
■ For instance, tutor designers tend to use principle
based hints rather than concrete hints in tutor problems that have brief problem scenarios
Knowledge discovery2) Structure discovery (VI)
● Social Network Analysis○ Models are developed of the relationships and
interactions between individual actors, as well as the
patterns that emerge from those relationships and interactions
○ Examples
■ Understanding the differences between effective and ineffective project groups [Kay2006]
■ How students’ communication behaviors change over time [Haythornthwaite2001]
■ How students’ positions in a social network relate
to their perception of being part of a learning community [Dawson2008]
Knowledge discovery2) Structure discovery (VII)
● Domain structure discovery○ Consists of finding the structure of knowledge in an
educational domain (e.g., how specific content maps to specific knowledge components or skills, across students)
○ This could consist of mapping problems in educational software to specific knowledge components, in order to group the problems effectively for latent knowledge
estimation and problem selection [Koedinger2006], or could consist of mapping test items to skills [Tatsuoka1995]
Knowledge discovery2) Structure discovery (VIII)
● WEKA contains “clusterers” for finding groups of similar instances in a dataset
● Implemented schemes are:○ k-Means, EM, Cobweb, X-means, FarthestFirst
● Clusters can be visualized and compared to “true” clusters (if given)
● Evaluation based on loglikelihood if clustering scheme produces a probability distribution
Knowledge discovery3) Relationship mining
● Discover relationships between variables in a data set with a large number of variables
● It has historically been the most common category of EDM research [Baker2009]
● It may take the form of attempting to find out which variables are most strongly associated with a single variable of particular interest
● Or may take the form of attempting to discover which relationships between any two variables are strongest
Knowledge discovery3) Relationship mining (II)
● There are four types of relationship mining○ Association rule mining○ Correlation mining○ Sequential pattern mining○ Causal data mining
Knowledge discovery3) Relationship mining (III)
● Association rule mining○ The goal is to find if-then rules of the form that if some
set of variable values is found, another variable will generally have a specific value
○ For instance, [BenNaim2009] used association rule mining to find patterns of successful student performance in an engineering simulation, to make better suggestions to students having difficulty about how they can improve their performance
Knowledge discovery3) Relationship mining (IV)
● Correlation mining○ The goal is to find positive or negative linear
correlations between variables (using post-hoc corrections or dimensionality reduction methods when appropriate to avoid finding spurious relationships)
○ An example can be found in [Baker2009], where correlations were computed between a range of features of the design of intelligent tutoring system lessons and students’ prevalence of gaming the system
Knowledge discovery3) Relationship mining (V)
● Sequential pattern mining○ The goal is to find temporal associations between
events
○ One successful use of this approach was work by
[Perera2009], to determine what path of student collaboration behaviors leads to a more successful eventual group project
Knowledge discovery3) Relationship mining (VI)
● Causal data mining○ The goal is to find whether one event (or observed
construct) was the cause of another event (or observed construct)
○ For example to predict which factors will lead a student to do poorly in a class [Fancsali2012]
Knowledge discovery3) Relationship mining (VII)
● WEKA contains an implementation of the Apriori algorithm for learning association rules○ Works only with discrete data
● Can identify statistical dependencies between groups of attributes:○ milk, butter bread, eggs (with confidence 0.9 and
support 2000)
● Apriori can compute all rules that have a given minimum support and exceed a given confidence
Knowledge discovery3) Relationship mining (VIII)
Knowledge discovery4) Attribute selection
● Panel that can be used to investigate which (subsets of) attributes are the most predictive ones
● Attribute selection methods contain two parts:○ A search method: best-first, forward selection,
random, exhaustive, genetic algorithm, ranking
○ An evaluation method: correlation-based, wrapper, information gain, chi-squared, etc.
● Very flexible: WEKA allows (almost) arbitrary combinations of these two
Knowledge discovery4) Attribute selection (II)
Knowledge discovery4) Attribute selection (III)
References[Amershi2009] Amershi, S., Conati, C. (2009). Combining Unsupervised and Supervised Machine Learning to Build User Models for Exploratory Learning Environments. Journal of Educational Data Mining, 1(1), 71-81.
[BakerSiemens2014] Baker, R., and George Siemens. "Educational data mining and learning analytics." Cambridge Handbook of the Learning Sciences: (2014).
[BakerYacef2009] Baker, R.S.J.d., Yacef, K. (2009). The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining, 1 (1), 3-17
[Beal2006] Beal, C.R., Qu, L., & Lee, H. (2006). Classifying learner engagement through integration of multiple data sources. Paper presented at the 21st National Conference on Artificial Intelligence (AAAI-2006), Boston, MA.
[CorbettAnderson1995] Corbett, A.T., Anderson, J.R. (1995). Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.
[Dawson2008] Dawson, S. (2008). A study of the relationship between student social networks and sense of community. Educational Technology & Society, 11(3), 224-238.
[Dekker2009] Dekker, G., Pechenizkiy, M., and Vleeshouwers, J. (2009). Predicting students drop out: A case study. Proceedings of the 2nd International Conference on Educational Data Mining, EDM'09, 41-50
[Fancsali2012] Fancsali, S. (2012) Variable Construction and Causal Discovery for Cognitive Tutor Log Data: Initial Results. Proceedings of the 5th International Conference on Educational Data Mining, 238-239.
[Feng2009] Feng, M., Heffernan, N., & Koedinger, K. (2009). Addressing the Assessment Challenge in an Intelligent Tutoring System that Tutors as it Assesses. User Modeling and User-Adapted Interaction, 19, 243-266
[Ferguson2012] Ferguson, R. (2012). The State Of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01, Knowledge Media Institute, The Open University, UK. http://kmi.open.ac.uk/publications/techreport/kmi-12-01
[Ferguson2014] Learning analytics FAQs [Online]. URL: http://www.slideshare.net/R3beccaF/learning-analytics-fa-qs
[Haythornthwaite2001] Haythornthwaite, C. (2001). Exploring Multiplexity: Social Network Structures in a ComputerSupported Distance Learning Class. The Information Society: An International Journal, 17 (3), 211-226.
[Kay2006] Kay, J., Maisonneuve, N., Yacef, K., Reimann, P. (2006) The Big Five and Visualisations of Team Work Activity. Proceedings of the International Conference on Intelligent Tutoring Systems, 197 – 206.
References (II)[Koedinger2006] Koedinger, K. R., & Corbett, A. T. (2006). Cognitive Tutors: Technology bringing learning science to the classroom. In K. Sawyer (Ed.) The Cambridge Handbook of the Learning Sciences (pp. 61-78). New York: Cambridge University Press.
[MingMing2012] Ming, N.C., Ming, V.L. (2012). Predicting Student Outcomes from Unstructured Data. Proceedings of the 2nd International Workshop on Personalization Approaches in Learning Environments, 11-16.
[Pavlik2009] Pavlik, P.I., Cen, H., Koedinger, K.R. (2009). Performance Factors Analysis -- A New Alternative to Knowledge Tracing. Proceedings of AIED2009.
[Perera2009] Perera, D., Kay, J., Koprinska, I., Yacef, K., and Zaiane, O.R. (2009). Clustering and Sequential Pattern Mining of Online Collaborative Learning Data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759-772
[RomeroVentura2010]Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state-ofthe-art. IEEE Transaction on Systems, Man and Cybernetics, part C: Applications and Reviews, 40(6), 610–618
[Sabourin2011] Sabourin, J., Rowe, J., Mott, B., Lester, J. (2011). When Off-Task in On-Task: The Affective Role of Off-Task Behavior in Narrative-Centered Learning Environments. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 534-536.
[SiemensBaker2012] Siemens, G., Baker, R.S.J.d. (2012). Learning Analytics and Educational Data Mining: Towards Communication and Collaboration. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge.
[Tatsuoka1995] Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment, 327–359. Hillsdale NJ: Erlbaum
[Verbert2011] Dataset-driven research to improve TEL recommender systems [Online]. URL: http://www.slideshare.net/kverbert/datasetdriven-research-to-improve-tel-recommender-systems
Data Analytics process in Learning and Academic
Analytics projects
Day 3: Data processing
Alex Rayón [email protected]
DeustoTech Learning – Deusto Institute of Technology – University of DeustoAvda. Universidades 24, 48007 Bilbao, Spain
www.deusto.es