If you can't read please download the document
Upload
francis-pieraut
View
630
Download
1
Embed Size (px)
DESCRIPTION
This is the slides of my master defense; 17 april 2003 subject: "High capacity neural network optimization problems: study & solutions exploration"
Citation preview
2. Plan
3. Learning algorithm: Neural Network
4. sortie zcible t t 1 t k y 1 x i x D y N w kj w ij x 1 Neural Networks and capacity P(c i |x i ) P(c i |x i ) y 2 y j z 1 Z k 5. 6. High/huge capacity Neural Network y 1 y 2 y 2 7. Constraints
8. Errors :Optimization Inefficiency of High Capacity Neural Networks 9. CPU time: Optimization Inefficiency of High Capacity Neural Networks 10. Is this inefficiency normal?
11. sortie zcible t z 1 Z k t 1 t k y 1 x i x D y N w kj w ij x 1 Neural Networks and equations y 2 y j 12. Learning process is slowing down for non-linear relationships 13. Solutions space of a N+K Neurones Neural Network Solution space of a NNeurones Neural Network Solutions space 14. Similar Solutions Initial State Example 5 iterations3 iterations 15. Optimisation problems
16. sortie zcible t z 1 Z k t 1 t k y 1 x i x D y N w jk w ij x 1 Neural Networks Optimization Problems
y 2 y j 17. Explored solutions
18. Incremental Neural Networks : first approach 19. Incremental Neural Networks : first approach (fix weights optimisation) 20. Hypothesis: Incremental NN OK Incremental NN Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution 21. Incremental Neural Networks (1): results 22. Why it doesnt work? (critical points) 23. 24. 25. Incremental Neural Network : second approach (add hidden layers) z 1 z 2 x 1 x 2 z 1 z 2 y 1 x 1 x 2 y 2 y 3 y 4 26. Cost function curve shape 27. Hypothesis: Incremental NN (add layers) OK Incremental NN (add layers)Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution 28. Incremental Neural Network (2): results 29. Uncoupled architecture 30. Hypothesis: Uncoupled Architecture OK Removed Decoupled architecture Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution 31. In efficiency of high capacity Neural Networks (CPU time) 32. Efficiency of High capacity Neural Network: decoupled architecture 33. Hypothesis: Partial Parameters optimization OK Opt. partie Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution 34. Neural Networks with partial parameters optimization: results All parametersoptimization Max sensitivity optimization 35. Why predicting parameters? (observation) poque Valeurs 36. Hypothesis * Benefit: reduce # iterations by predicting values based on history Parameter prediction Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution 37. Prediction : Quadratic extrapolation 38. Prediction : Learning rate increase 39. Contributions
40. Futur works
41. Conclusion
42. Any Questions?? 43. Exemple :solution linaire 44. Exemple :solution hautement non-linaire 45. Slection des connections influenant le plus le cot 46. Slection des connections influenant le plus lerreur T = 1 S = 0 T = 0 S = 1 T = 0 S = 0.1 T = 0 S = 0.1 47. Observation: idealized behavior of the ratio time