Upload
amyhaynes
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Machine Learning Theory
MariaFlorina Balcan
Lecture 1, Jan. 12th 2010
2
Image Classification
Document Categorization
Speech Recognition Branch Prediction
Protein Classification
Spam DetectionFraud Detection
Machine Learning
Playing Games Computational Advertising
• what kinds of tasks we can hope to learn, and from what kind of data
Goals of Machine Learning TheoryDevelop and analyze models to
understand:
• what types of guarantees might we hope to achieve
• prove guarantees for practically successful algs (when will they succeed, how long will they take?);
• Algorithms
Interesting connections to other areas including:
• Combinatorial Optimization
• Probability & Statistics • Game Theory
• Information Theory• Complexity Theory
• develop new algs that provably meet desired criteria
4
Example: Supervised Classification
Goal: use emails seen so far to produce good prediction rule for future data.
Not spam spam
Decide which emails are spam and which are important.
Supervised classification
5
example label
Reasonable RULES:
Predict SPAM if unknown AND (money OR pills)
Predict SPAM if 2money + 3pills –5 known > 0
Represent each message by features. (e.g., keywords, spelling, etc.)
Example: Supervised Classification
+

+++



Linearly separable
6
Two Main Aspects of Supervised Learning
Algorithm Design. How to optimize?
Automatically generate rules that do well on observed data.
Confidence Bounds, Generalization Guarantees, Sample Complexity
Confidence for rule effectiveness on future data.
Well understood for passive supervised learning.
7
• SemiSupervised Learning
Using cheap unlabeled data in addition to labeled data.
• Active Learning
The algorithm interactively asks for labels of informative examples.
Other Protocols for Supervised Learning
Theoretical understanding severely lacking until a couple of years ago. Lots of progress recently. We will cover some of these.
• Learning with Membership Queries
• Statistical Query Learning
Structure of the Class
• Simple algos and hardness results for supervised learning.
• Classic, state of the art algorithms: AdaBoost and SVM.
• Basic models for supervised learning: PAC and SLT.
• Standard Sample Complexity Results (VC dimension)• Weaklearning vs. Stronglearning
Structure of the Class
• Classification noise and the StatisticalQuery model
• Incorporating Unlabeled Data in the Learning Process.
• Reinforcement Learning
• Learning Real Valued Functions
• Modern Sample Complexity Results• Rademacher Complexity
• Margin analysis of Boosting and SVM
• Incorporating Interaction in the Learning Process: • Active Learning• Learning with Membership Queries
Admin
• Course web page:
• 6 hwk assignments. Exercises/problems (pencilandpaper problemsolving variety).
• Small project: explore a theoretical question, try some experiments, or read a paper and explain the idea. Short writeup and possibly presentation. Small groups ok.
• Takehome exam.
http://www.cc.gatech.edu/~ninamf/ML10/
[60%]
[25%]
[15%]