A work on optimizers for binarized neural networks: a second order approach

Abstract

Optimization of Binarized Neural Networks (BNNs) relies on approximating the real-valued weights with their binarized representations. Current techniques for weight-updating uses the same optimizers as traditional Neural Networks (NNs). There has only been one effort to directly train the BNNs with bit-flips by using a raw first moment estimate of the gradients and comparing it against a threshold for deciding when to flip a weight (Bop). In this thesis, we iteratively improve this approach by drawing parallels to the Adam optimizer with the inclusion of a second raw moment estimate to normalize the average of the gradients before doing the comparison with a threshold (Bop2ndOrder). Additionally, we tested the effect of using a scheduler on the threshold value as an equivalent to a regularizer, along with bias-corrected and not corrected versions of the optimizer. The proposed optimizer was tested using three different architectures with CIFAR-10 and Imagenet2012; in both datasets this proved to converge faster, being more robust to changes of the hyper-parameters, and achieving better accuracies. Moreover, we also proposed a proof of concept Probabilistic Binary Optimizer (PBop) which treats each weight as loaded coins (Bernoulli distribution) proving that, even though the results are not on par with state-of-the-art, the concept is feasible for Image Classification although it requires a deep exploration of the effect of the scaler.