Intro to Machine LearningRecitation 1: Intro, KNN, K-Means
1
● Name: Yossi Adi and Felix Kreuk ● Number of assignments: 6
○ Programming (python) and theoretical exercises ○ Checked with the “Submit” system.
● % assignments, % test ● Office hours: ?? ● Piazza: ? ● Project ?
Administrative Stuff
2
Machine Learning is Everywhere
3
INTERNET & CLOUD
Image Classification Speech Recognition Language Translation Language Processing
Sentiment Analysis Recommendation
MEDICINE & BIOLOGY
Cancer Cell Detection Diabetic Grading Drug Discovery
MEDIA & ENTERTAINMENT
Video Captioning Video Search
Real Time Translation
SECURITY & DEFENCES
Face Detection Video Surveillance Satellite Imagery
AUTONOMOUS MACHINES
Pedestrian Detection Lane Tracking
Recognize Traffic Sign
Machine Learning
Herbert Alexander Simon:
“Learning is any process by which a system improves performance from experience.”
“Machine Learning is concerned with computer programs that automatically improve their performance through experience. “
4
Why use Machine Learning?
● Develop systems that can automatically adapt themselves ○ Personalised systems (facebook feed, google search)
● Discover new knowledge from large databases (cluster users) ● Market basket analysis (e.g. diapers and beer) ● Mimic human abilities in mundane tasks
○ Recognising handwritten characters ○ Speech recognition ○ Image annotation
● Develop systems that are too difficult to construct manually (if & else)
5
Why Now?
● Flood of data ● Computational power ● Research Boost (academic and industry)
6
What is Learning?
Learning = Improving with experience at some task
Improve over task - T
With respect to performance measure - P
Based on experience - E
7
Example - image classification
{cat, dog, …, car}
*Adapted from Stanford cs231n course presentations. 8
Example - image classification
{cat, dog, …, car}
*Adapted from Stanford cs231n course presentations. 9
Challenges: viewpoint
All pixels change when the camera moves!
*Adapted from Stanford cs231n course presentations. 10
Challenges: illumination
*Adapted from Stanford cs231n course presentations. 11
Challenges: Deformation
*Adapted from Stanford cs231n course presentations. 12
Challenges: Occlusion
*Adapted from Stanford cs231n course presentations. 13
Challenges: Background Clutter
*Adapted from Stanford cs231n course presentations. 14
Challenges: Intraclass variation
*Adapted from Stanford cs231n course presentations. 15
Learning Schemes
● Supervised Learning ● Examples are labeled
● Unsupervised ● No labels
● Semi-Supervised ● Partially labeled
● Reinforcement Learning ● Reward
16
Outputs
● Classification: one over K classes ● Object classification, text classification, etc.
● Regression: predict real valued vector ● Stock values, height prediction, etc.
● Structured: predict complex outputs with structure ● Parsing trees, ASR, translation, etc.
17
Separators and Generalisation
Which one would you choose?
18
Separators and Generalisation
19
Separators and Generalisation
Linear case - two mistakes
Non Linear case - zero mistakes
20
Terminology
● Target function: ● In classification: ● In regression:
● Hypothesis: A proposed function , an approximation of . ● Hypothesis space: The space of all hypotheses that can, in principle, be
output by the learning algorithm. ● Model: A function . The output of our learning algorithm. ● Loss: an evaluation metric:
Y =�1, 2, · · · , k
<latexit sha1_base64="y4mvVQD7XNlEXlh4MjctA2YSyXc=">AAACB3icbVBNS8NAEN34WetX1KMHF4vgoZSkCOpBKHrxWMHYShPKZrNtl252w+5GKKFHL/4VLx5UvPoXvPlv3KY5aOuDgcd7M8zMCxNGlXacb2thcWl5ZbW0Vl7f2Nzatnd275RIJSYeFkzIdogUYZQTT1PNSDuRBMUhI61weDXxWw9EKir4rR4lJIhRn9MexUgbqWsf3MML6Ie072duFdar0MeR0KoKh7k47toVp+bkgPPELUgFFGh27S8/EjiNCdeYIaU6rpPoIENSU8zIuOyniiQID1GfdAzlKCYqyPJHxvDIKBHsCWmKa5irvycyFCs1ikPTGSM9ULPeRPzP66S6dxZklCepJhxPF/VSBrWAk1RgRCXBmo0MQVhScyvEAyQR1ia7sgnBnX15nnj12nnNuTmpNC6LNEpgHxyCY+CCU9AA16AJPIDBI3gGr+DNerJerHfrY9q6YBUze+APrM8fyGeXgQ==</latexit><latexit sha1_base64="y4mvVQD7XNlEXlh4MjctA2YSyXc=">AAACB3icbVBNS8NAEN34WetX1KMHF4vgoZSkCOpBKHrxWMHYShPKZrNtl252w+5GKKFHL/4VLx5UvPoXvPlv3KY5aOuDgcd7M8zMCxNGlXacb2thcWl5ZbW0Vl7f2Nzatnd275RIJSYeFkzIdogUYZQTT1PNSDuRBMUhI61weDXxWw9EKir4rR4lJIhRn9MexUgbqWsf3MML6Ie072duFdar0MeR0KoKh7k47toVp+bkgPPELUgFFGh27S8/EjiNCdeYIaU6rpPoIENSU8zIuOyniiQID1GfdAzlKCYqyPJHxvDIKBHsCWmKa5irvycyFCs1ikPTGSM9ULPeRPzP66S6dxZklCepJhxPF/VSBrWAk1RgRCXBmo0MQVhScyvEAyQR1ia7sgnBnX15nnj12nnNuTmpNC6LNEpgHxyCY+CCU9AA16AJPIDBI3gGr+DNerJerHfrY9q6YBUze+APrM8fyGeXgQ==</latexit><latexit sha1_base64="y4mvVQD7XNlEXlh4MjctA2YSyXc=">AAACB3icbVBNS8NAEN34WetX1KMHF4vgoZSkCOpBKHrxWMHYShPKZrNtl252w+5GKKFHL/4VLx5UvPoXvPlv3KY5aOuDgcd7M8zMCxNGlXacb2thcWl5ZbW0Vl7f2Nzatnd275RIJSYeFkzIdogUYZQTT1PNSDuRBMUhI61weDXxWw9EKir4rR4lJIhRn9MexUgbqWsf3MML6Ie072duFdar0MeR0KoKh7k47toVp+bkgPPELUgFFGh27S8/EjiNCdeYIaU6rpPoIENSU8zIuOyniiQID1GfdAzlKCYqyPJHxvDIKBHsCWmKa5irvycyFCs1ikPTGSM9ULPeRPzP66S6dxZklCepJhxPF/VSBrWAk1RgRCXBmo0MQVhScyvEAyQR1ia7sgnBnX15nnj12nnNuTmpNC6LNEpgHxyCY+CCU9AA16AJPIDBI3gGr+DNerJerHfrY9q6YBUze+APrM8fyGeXgQ==</latexit>
Y = Rn<latexit sha1_base64="9LkLj5qkmtpvoPesQVkZbXaqbmI=">AAAB+HicbVBNS8NAFHypX7V+RT16WSyCp5KIoB6EohePVYyttLFstpt26WYTdjeFEvpPvHhQ8epP8ea/cdPmoK0DC8PMe7zZCRLOlHacb6u0tLyyulZer2xsbm3v2Lt7DypOJaEeiXksWwFWlDNBPc00p61EUhwFnDaD4XXuN0dUKhaLez1OqB/hvmAhI1gbqWvbj+gSdSKsB0GQ3U2ejFR1as4UaJG4BalCgUbX/ur0YpJGVGjCsVJt10m0n2GpGeF0UumkiiaYDHGftg0VOKLKz6bJJ+jIKD0UxtI8odFU/b2R4UipcRSYyTyjmvdy8T+vnerw3M+YSFJNBZkdClOOdIzyGlCPSUo0HxuCiWQmKyIDLDHRpqyKKcGd//Ii8U5qFzXn9rRavyraKMMBHMIxuHAGdbiBBnhAYATP8ApvVma9WO/Wx2y0ZBU7+/AH1ucPCZeS1A==</latexit><latexit sha1_base64="9LkLj5qkmtpvoPesQVkZbXaqbmI=">AAAB+HicbVBNS8NAFHypX7V+RT16WSyCp5KIoB6EohePVYyttLFstpt26WYTdjeFEvpPvHhQ8epP8ea/cdPmoK0DC8PMe7zZCRLOlHacb6u0tLyyulZer2xsbm3v2Lt7DypOJaEeiXksWwFWlDNBPc00p61EUhwFnDaD4XXuN0dUKhaLez1OqB/hvmAhI1gbqWvbj+gSdSKsB0GQ3U2ejFR1as4UaJG4BalCgUbX/ur0YpJGVGjCsVJt10m0n2GpGeF0UumkiiaYDHGftg0VOKLKz6bJJ+jIKD0UxtI8odFU/b2R4UipcRSYyTyjmvdy8T+vnerw3M+YSFJNBZkdClOOdIzyGlCPSUo0HxuCiWQmKyIDLDHRpqyKKcGd//Ii8U5qFzXn9rRavyraKMMBHMIxuHAGdbiBBnhAYATP8ApvVma9WO/Wx2y0ZBU7+/AH1ucPCZeS1A==</latexit><latexit sha1_base64="9LkLj5qkmtpvoPesQVkZbXaqbmI=">AAAB+HicbVBNS8NAFHypX7V+RT16WSyCp5KIoB6EohePVYyttLFstpt26WYTdjeFEvpPvHhQ8epP8ea/cdPmoK0DC8PMe7zZCRLOlHacb6u0tLyyulZer2xsbm3v2Lt7DypOJaEeiXksWwFWlDNBPc00p61EUhwFnDaD4XXuN0dUKhaLez1OqB/hvmAhI1gbqWvbj+gSdSKsB0GQ3U2ejFR1as4UaJG4BalCgUbX/ur0YpJGVGjCsVJt10m0n2GpGeF0UumkiiaYDHGftg0VOKLKz6bJJ+jIKD0UxtI8odFU/b2R4UipcRSYyTyjmvdy8T+vnerw3M+YSFJNBZkdClOOdIzyGlCPSUo0HxuCiWQmKyIDLDHRpqyKKcGd//Ii8U5qFzXn9rRavyraKMMBHMIxuHAGdbiBBnhAYATP8ApvVma9WO/Wx2y0ZBU7+/AH1ucPCZeS1A==</latexit>
h<latexit sha1_base64="lt6MdrejMYgtNBMJ4G/0FERlp4M=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaw16l6tbcGcgy8QpShQKNXuWr209YFqM0TFCtO56bmiCnynAmcFLuZhpTykZ0gB1LJY1RB/ns0Ak5tUqfRImyJQ2Zqb8nchprPY5D2xlTM9SL3lT8z+tkJroKci7TzKBk80VRJohJyPRr0ucKmRFjSyhT3N5K2JAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA8HYy/</latexit><latexit sha1_base64="lt6MdrejMYgtNBMJ4G/0FERlp4M=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaw16l6tbcGcgy8QpShQKNXuWr209YFqM0TFCtO56bmiCnynAmcFLuZhpTykZ0gB1LJY1RB/ns0Ak5tUqfRImyJQ2Zqb8nchprPY5D2xlTM9SL3lT8z+tkJroKci7TzKBk80VRJohJyPRr0ucKmRFjSyhT3N5K2JAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA8HYy/</latexit><latexit sha1_base64="lt6MdrejMYgtNBMJ4G/0FERlp4M=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaw16l6tbcGcgy8QpShQKNXuWr209YFqM0TFCtO56bmiCnynAmcFLuZhpTykZ0gB1LJY1RB/ns0Ak5tUqfRImyJQ2Zqb8nchprPY5D2xlTM9SL3lT8z+tkJroKci7TzKBk80VRJohJyPRr0ucKmRFjSyhT3N5K2JAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA8HYy/</latexit>
f<latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit>
` : R⇥ R ! R+<latexit sha1_base64="CwMi5ZjwYsMTbYGBxpegtchb5sc=">AAACG3icbVDLSsNAFL2pr1pfUZduBosgCCVRwceq6MZlFWMLTSyT6aQdOnkwMxFK6Ie48VfcuFBxJbjwb5y0XfThgYEz597Lvef4CWdSWdavUVhYXFpeKa6W1tY3NrfM7Z0HGaeCUIfEPBYNH0vKWUQdxRSnjURQHPqc1v3edV6vP1EhWRzdq35CvRB3IhYwgpWWWuaJSzm/RG6IVdf3s7sBchULqZxW4onv41HLLFsVawg0T+wxKcMYtZb57bZjkoY0UoRjKZu2lSgvw0Ixwumg5KaSJpj0cIc2NY2wPsDLhuYG6EArbRTEQr9IoaE6OZHhUMp+6OvO/EY5W8vF/2rNVAXnXsaiJFU0IqNFQcqRdpsnhdpMUKJ4XxNMBNO3ItLFAhOl8yzpEOxZy/PEOa5cVKzb03L1apxGEfZgHw7BhjOowg3UwAECz/AK7/BhvBhvxqfxNWotGOOZXZiC8fMH26yhcQ==</latexit><latexit sha1_base64="CwMi5ZjwYsMTbYGBxpegtchb5sc=">AAACG3icbVDLSsNAFL2pr1pfUZduBosgCCVRwceq6MZlFWMLTSyT6aQdOnkwMxFK6Ie48VfcuFBxJbjwb5y0XfThgYEz597Lvef4CWdSWdavUVhYXFpeKa6W1tY3NrfM7Z0HGaeCUIfEPBYNH0vKWUQdxRSnjURQHPqc1v3edV6vP1EhWRzdq35CvRB3IhYwgpWWWuaJSzm/RG6IVdf3s7sBchULqZxW4onv41HLLFsVawg0T+wxKcMYtZb57bZjkoY0UoRjKZu2lSgvw0Ixwumg5KaSJpj0cIc2NY2wPsDLhuYG6EArbRTEQr9IoaE6OZHhUMp+6OvO/EY5W8vF/2rNVAXnXsaiJFU0IqNFQcqRdpsnhdpMUKJ4XxNMBNO3ItLFAhOl8yzpEOxZy/PEOa5cVKzb03L1apxGEfZgHw7BhjOowg3UwAECz/AK7/BhvBhvxqfxNWotGOOZXZiC8fMH26yhcQ==</latexit><latexit sha1_base64="CwMi5ZjwYsMTbYGBxpegtchb5sc=">AAACG3icbVDLSsNAFL2pr1pfUZduBosgCCVRwceq6MZlFWMLTSyT6aQdOnkwMxFK6Ie48VfcuFBxJbjwb5y0XfThgYEz597Lvef4CWdSWdavUVhYXFpeKa6W1tY3NrfM7Z0HGaeCUIfEPBYNH0vKWUQdxRSnjURQHPqc1v3edV6vP1EhWRzdq35CvRB3IhYwgpWWWuaJSzm/RG6IVdf3s7sBchULqZxW4onv41HLLFsVawg0T+wxKcMYtZb57bZjkoY0UoRjKZu2lSgvw0Ixwumg5KaSJpj0cIc2NY2wPsDLhuYG6EArbRTEQr9IoaE6OZHhUMp+6OvO/EY5W8vF/2rNVAXnXsaiJFU0IqNFQcqRdpsnhdpMUKJ4XxNMBNO3ItLFAhOl8yzpEOxZy/PEOa5cVKzb03L1apxGEfZgHw7BhjOowg3UwAECz/AK7/BhvBhvxqfxNWotGOOZXZiC8fMH26yhcQ==</latexit>
f<latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit>
21
t : X ! Y<latexit sha1_base64="DJMpgH1mE0VRzKAYzxwVQroWcWo=">AAAB8HicbVDLSgMxFL1TX7W+qi7dBIvgqkxF8LEqunFZwbHVdiiZNNOGZjJDckcoQ//CjQsVt36OO//GtJ2Fth4IHM65l9xzgkQKg6777RSWlldW14rrpY3Nre2d8u7evYlTzbjHYhnrVkANl0JxDwVK3ko0p1EgeTMYXk/85hPXRsTqDkcJ9yPaVyIUjKKVHvGStEgHY/LQLVfcqjsFWSS1nFQgR6Nb/ur0YpZGXCGT1Jh2zU3Qz6hGwSQflzqp4QllQ9rnbUsVjbjxs+nFY3JklR4JY22fQjJVf29kNDJmFAV2MqI4MPPeRPzPa6cYnvuZUEmKXLHZR2EqiY04iU96QnOGcmQJZVrYWwkbUE0Z2pJKtoTafORF4p1UL6ru7WmlfpW3UYQDOIRjqMEZ1OEGGuABAwXP8ApvjnFenHfnYzZacPKdffgD5/MHfquPrw==</latexit><latexit sha1_base64="DJMpgH1mE0VRzKAYzxwVQroWcWo=">AAAB8HicbVDLSgMxFL1TX7W+qi7dBIvgqkxF8LEqunFZwbHVdiiZNNOGZjJDckcoQ//CjQsVt36OO//GtJ2Fth4IHM65l9xzgkQKg6777RSWlldW14rrpY3Nre2d8u7evYlTzbjHYhnrVkANl0JxDwVK3ko0p1EgeTMYXk/85hPXRsTqDkcJ9yPaVyIUjKKVHvGStEgHY/LQLVfcqjsFWSS1nFQgR6Nb/ur0YpZGXCGT1Jh2zU3Qz6hGwSQflzqp4QllQ9rnbUsVjbjxs+nFY3JklR4JY22fQjJVf29kNDJmFAV2MqI4MPPeRPzPa6cYnvuZUEmKXLHZR2EqiY04iU96QnOGcmQJZVrYWwkbUE0Z2pJKtoTafORF4p1UL6ru7WmlfpW3UYQDOIRjqMEZ1OEGGuABAwXP8ApvjnFenHfnYzZacPKdffgD5/MHfquPrw==</latexit><latexit sha1_base64="DJMpgH1mE0VRzKAYzxwVQroWcWo=">AAAB8HicbVDLSgMxFL1TX7W+qi7dBIvgqkxF8LEqunFZwbHVdiiZNNOGZjJDckcoQ//CjQsVt36OO//GtJ2Fth4IHM65l9xzgkQKg6777RSWlldW14rrpY3Nre2d8u7evYlTzbjHYhnrVkANl0JxDwVK3ko0p1EgeTMYXk/85hPXRsTqDkcJ9yPaVyIUjKKVHvGStEgHY/LQLVfcqjsFWSS1nFQgR6Nb/ur0YpZGXCGT1Jh2zU3Qz6hGwSQflzqp4QllQ9rnbUsVjbjxs+nFY3JklR4JY22fQjJVf29kNDJmFAV2MqI4MPPeRPzPa6cYnvuZUEmKXLHZR2EqiY04iU96QnOGcmQJZVrYWwkbUE0Z2pJKtoTafORF4p1UL6ru7WmlfpW3UYQDOIRjqMEZ1OEGGuABAwXP8ApvjnFenHfnYzZacPKdffgD5/MHfquPrw==</latexit>
Terminology: Empirical Risk Minimization
Goal: to minimize:
S = {(x1,y1), . . . , (xm,ym)}
Rho is unknown, so we use a set of examples:
E(x,y)⇠⇢ [`(y, y(x))]
We would like to minimize:
w⇤ = argminw
1
m
mX
i=1
`(yi, yw(xi)) +�
2⌦(w)
22
Terminology
● Training set ● Used to train the classifier
● Validation set ● Used to tune hyper-parameters
● Test set ● Not seen during training, used to evaluate our classifier
● Epoch: one “pass” over the training set
23
Training Process
● Training loop: ● Modify predictor function according to the training set ● Evaluate the loss on the validation set
● Evaluate the loss on test set ● Output predictor function
24
(which performs “well” on the test set according to the loss function)
Terminology: overfitting and underfitting
25# Iterations
Loss
K Nearest Neighbours
26
K Nearest Neighbours (KNN)
● Our first machine learning algorithm ● One of the simplest algorithms ● Algorithm:
● Given a test example x ● Define a metric d:
● Non-negative ● Identity ● Symmetry ● Triangle inequality
● Find the k closest examples to x according to d ● Predict the majority class
27
K Nearest Neighbours (KNN) - Properties
● Supervised ● Lazy ● Need to specify K ● Non-Linear ● Different metrics will output different boundaries ● New examples directly change the classifier ● Every dimension contributes equally ● Sensitive to outliers
28
K Means
30
K Means
● The k-means algorithm is an iterative method for clustering a set of N points in to K clusters
● Algorithm: ● Define a metric d ● Initialise K centroids ● Repeat until convergence:
● Assign each point to the closest centroid according to d
● Update each centroid to be the mean of the points in its group
31
K Means - Properties● Unsupervised ● Need to specify K ● Different centroids and metrics will output different boundaries ● Assumptions: spherical, same size and density ● Sensitive to outliers
32
Questions?
34