Exploring optimizers in Deep Learning

Optimizer algorithms are the foundation of how to train Intelligent Systems. They do so by updating the learnable parameters of a model so that its performance (usually measured with a Objective/Error function) is optimal. These algorithms influence the model’s learning process and its output in a crucial way.

Every state-of-the-art library deep learning library contains many implementations of such algorithms, which are used as “black-box” optimizers, since their documentations lack an explanation of their strengths and weakness. Therefore, a lot is left unexplored with respect to the best suitable optimizer for the task at hand.