An Adiabatic Method to Train Binarized Artificial Neural Networks

Abstract An artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (reLU) functions, etc. Synapses connect the neuron outputs to their inputs with tunable real-valued weights. The most resource-demanding operations in realizing such neural networks are the multiplication and accumulate (MAC) operations that compute the dot product be- tween real-valued outputs from neurons and the synapses weights. The efficiency of neural networks can be drastically enhanced if the neuron outputs and/or the weights can be trained to take binary values ±1 only, for which the MAC can be replaced by the simple XOR operations. In this paper, we demonstrate an adiabatic training method that can successfully binarize the dense neural networks and the convolutional neural networks without modification in terms network structure and with very minimal change in training algorithms. This adiabatic training method is tested in the following four tasks: the recognition of hand-writing numbers using a usual dense network, the cat-dog recog- nition and the audio recognition using a convolutional neural networks, the image recognition with 10 classes (CIFAR-10) using ResNet20 and VGG-Small networks. In all tasks, the performance of the binary neural networks trained by the adiabatic method are almost identical to the networks trained using the conventional reLU or Sigmoid activations with real-valued activations and weights. This adiabatic method can be easily applied to binarize different types of networks, and will increase the computational efficiency considerably and greatly simplify the deployment of neural networks.