21 - Deep Learning - Regularization Part 5 2020

<p>Deep Learning - Regularization Part 5</p>
<p>This video discusses multi-task learning.</p>
<p>Video References:<br />
<a href="https://www.youtube.com/channel/UCSHZKyawb77ixDdsGog4iWA">Lex Fridman&#39;s Channel</a></p>
<p>Further Reading:<br />
<a href="https://www.sciencedirect.com/science/article/pii/S093938891830120X">A gentle Introduction to Deep Learning</a></p>
<p>Links:</p>
<p><a href="https://www.springer.com/us/book/9780387310732">Link</a> - for details on Maximum A Posteriori estimation and the bias-variance&nbsp;decomposition<br />
<a href="https://arxiv.org/abs/1206.5533">Link</a> - for a comprehensive text about practical recommendations for&nbsp;regularization<br />
<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.207.2059&amp;rep=rep1&amp;type=pdf">Link</a> - the paper about calibrating the variances</p>
<p>References:<br />
[1] Sergey Ioffe and Christian Szegedy. &ldquo;Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift&rdquo;. In: Proceedings of The 32nd International Conference on Machine Learning. 2015, pp. 448&ndash;456.<br />
[2] Jonathan Baxter. &ldquo;A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling&rdquo;. In: Machine Learning 28.1 (July 1997), pp. 7&ndash;39.<br />
[3] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Secaucus, NJ, USA: Springer-Verlag New York, Inc., 2006.<br />
[4] Richard Caruana. &ldquo;Multitask Learning: A Knowledge-Based Source of&nbsp;Inductive Bias&rdquo;. In:&nbsp;Proceedings of the Tenth International Conference on Machine Learning.&nbsp;Morgan Kaufmann, 1993, pp. 41&ndash;48.<br />
[5] Andre Esteva, Brett Kuprel, Roberto A Novoa, et al. &ldquo;Dermatologist-level&nbsp;classification of skin cancer with deep neural networks&rdquo;. In: Nature&nbsp;542.7639 (2017), pp. 115&ndash;118.<br />
[6] C. Ding, C. Xu, and D. Tao. &ldquo;Multi-Task Pose-Invariant Face Recognition&rdquo;.&nbsp;In: IEEE Transactions on Image Processing 24.3 (Mar. 2015), pp. 980&ndash;993.<br />
[7] Li Wan, Matthew Zeiler,&nbsp;Sixin Zhang, et al. &ldquo;Regularization of neural networks using drop connect&rdquo;. In:&nbsp;Proceedings of the 30th International Conference on Machine Learning (ICML-2013), pp. 1058&ndash;1066.<br />
[8] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, et al. &ldquo;Dropout: a&nbsp;simple way to prevent neural networks from overfitting.&rdquo; In:&nbsp;Journal of Machine Learning Research 15.1 (2014), pp. 1929&ndash;1958.<br />
[9] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification. John Wiley&nbsp;and Sons, inc., 2000.<br />
[10] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning.&nbsp;http://www.deeplearningbook.org. MIT Press, 2016.<br />
[11] Yuxin Wu and Kaiming He. &ldquo;Group normalization&rdquo;. In:&nbsp;arXiv preprint arXiv:1803.08494 (2018).<br />
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al. &ldquo;Delving deep into&nbsp;rectifiers: Surpassing human-level performance on imagenet classification&rdquo;.&nbsp;In: Proceedings of the IEEE international conference on computer vision.&nbsp;2015, pp. 1026&ndash;1034.<br />
[13] D Ulyanov, A Vedaldi, and VS Lempitsky.&nbsp;Instance normalization: the missing ingredient for fast stylization. CoRR abs/1607.0 [14] G&uuml;nter Klambauer, Thomas Unterthiner, Andreas Mayr, et al.&nbsp;&ldquo;Self-Normalizing Neural Networks&rdquo;. In:&nbsp;Advances in Neural Information Processing Systems (NIPS).&nbsp;Vol. abs/1706.02515. 2017. arXiv: 1706.02515.<br />
[15] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. &ldquo;Layer&nbsp;normalization&rdquo;. In: arXiv preprint arXiv:1607.06450 (2016).<br />
[16] Nima Tajbakhsh, Jae Y Shin, Suryakanth R Gurudu, et al. &ldquo;Convolutional&nbsp;neural networks for medical image analysis: Full training or fine tuning?&rdquo; In:&nbsp;IEEE transactions on medical imaging 35.5 (2016), pp. 1299&ndash;1312.<br />
[17] Yoshua Bengio. &ldquo;Practical recommendations for gradient-based training of&nbsp;deep architectures&rdquo;. In: Neural networks: Tricks of the trade. Springer,&nbsp;2012, pp. 437&ndash;478.<br />
[18] Chiyuan Zhang, Samy Bengio, Moritz Hardt, et al. &ldquo;Understanding deep&nbsp;learning requires rethinking generalization&rdquo;. In:&nbsp;arXiv preprint arXiv:1611.03530 (2016).<br />
[19] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, et al. &ldquo;How Does Batch&nbsp;Normalization Help Optimization?&rdquo; In: arXiv e-prints, arXiv:1805.11604&nbsp;(May 2018), arXiv:1805.11604. arXiv: 1805.11604 [stat.ML].<br />
[20] Tim Salimans and Diederik P Kingma. &ldquo;Weight Normalization: A Simple&nbsp;Reparameterization to Accelerate Training of Deep Neural Networks&rdquo;. In:&nbsp;Advances in Neural Information Processing Systems 29. Curran Associates,&nbsp;Inc., 2016, pp. 901&ndash;909.<br />
[21] Xavier Glorot and Yoshua Bengio. &ldquo;Understanding the difficulty of training&nbsp;deep feedforward neural networks&rdquo;. In:&nbsp;Proceedings of the Thirteenth International Conference on Artificial Intelligence 2010, pp. 249&ndash;256.<br />
[22] Zhanpeng Zhang, Ping Luo, Chen Change Loy, et al. &ldquo;Facial Landmark&nbsp;Detection by Deep Multi-task Learning&rdquo;. In:&nbsp;Computer Vision &ndash; ECCV 2014: 13th European Conference, Zurich, Switzerland, Cham: Springer International Publishing, 2014, pp. 94&ndash;108.</p>
<p>&nbsp;</p>