Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizing a probability distribution over the binary synapses. Both analytical and numerical results are presented. An algorithmic extension that allows to train discrete deep neural networks is also investigated.

Role of synaptic stochasticity in training low-precision neural networks

Baldassi, Carlo;Lucibello, Carlo;Saglietti, Luca;Zecchina, Riccardo
2018

Abstract

Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizing a probability distribution over the binary synapses. Both analytical and numerical results are presented. An algorithmic extension that allows to train discrete deep neural networks is also investigated.
2018
2018
Baldassi, Carlo; Gerace, Federica; Kappen, Hilbert J.; Lucibello, Carlo; Saglietti, Luca; Tartaglione, Enzo; Zecchina, Riccardo
File in questo prodotto:
File Dimensione Formato  
Baldassi.pdf

accesso aperto

Tipologia: Documento in Pre-print (Pre-print document)
Licenza: PUBBLICO DOMINIO
Dimensione 681.57 kB
Formato Adobe PDF
681.57 kB Adobe PDF Visualizza/Apri
baldassi2018.pdf

non disponibili

Descrizione: articolo
Tipologia: Pdf editoriale (Publisher's layout)
Licenza: NON PUBBLICO - Accesso privato/ristretto
Dimensione 271.21 kB
Formato Adobe PDF
271.21 kB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11565/4011427
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 17
  • ???jsp.display-item.citation.isi??? 14
social impact