Gonzalez Mendoza, MiguelSuárez Ramírez, Cuauhtémoc Daniel2022-01-102022-01-102020-11-212020-11Suarez Ramirez, C. D. (2020). A work on optimizers for binarized neural networks: a second order approach (Tesis de Maestría). Instituto Tecnológico y de Estudios Superiores de Monterrey. Recuperado de: https://hdl.handle.net/11285/643391https://hdl.handle.net/11285/643391https://orcid.org/0000-0001-6451-9109Optimization of Binarized Neural Networks (BNNs) relies on approximating the real-valued weights with their binarized representations. Current techniques for weight-updating uses the same optimizers as traditional Neural Networks (NNs). There has only been one effort to directly train the BNNs with bit-flips by using a raw first moment estimate of the gradients and comparing it against a threshold for deciding when to flip a weight (Bop). In this thesis, we iteratively improve this approach by drawing parallels to the Adam optimizer with the inclusion of a second raw moment estimate to normalize the average of the gradients before doing the comparison with a threshold (Bop2ndOrder). Additionally, we tested the effect of using a scheduler on the threshold value as an equivalent to a regularizer, along with bias-corrected and not corrected versions of the optimizer. The proposed optimizer was tested using three different architectures with CIFAR-10 and Imagenet2012; in both datasets this proved to converge faster, being more robust to changes of the hyper-parameters, and achieving better accuracies. Moreover, we also proposed a proof of concept Probabilistic Binary Optimizer (PBop) which treats each weight as loaded coins (Bernoulli distribution) proving that, even though the results are not on par with state-of-the-art, the concept is feasible for Image Classification although it requires a deep exploration of the effect of the scaler.TextoengopenAccesshttp://creativecommons.org/licenses/by-nc-nd/4.0INGENIERÍA Y TECNOLOGÍA::CIENCIAS TECNOLÓGICAS::TECNOLOGÍA DE LOS ORDENADORES::INTELIGENCIA ARTIFICIALTechnologyTesis de maestríahttps://orcid.org/0000-0003-3131-1330BinarizationBinarized Neural NetworksDeep LearningOptimizationOptimizerQuantization504827