tweaks to inhib model and delays

paper/sections/delays/Delays.md

@@ -1,24 +1,24 @@
 (delay-section)=
 ## Learning Delays
 
-This section introduces a simple method to solve the sound localization problem with only learnable delays
+This section introduces a simple method to solve the sound localization problem with only learnable delays.
 
 ### Introduction
 
-Studying the computational properties of axonal transmissions goes as far back as in \citealp{LAJ1948}. In this study, it was shown that with the right network setup, axonal delays can be utilized to transform a temporal cue to a spatial one for sound localization. This study was a leading study in terms of using delays explicitly to explain a neuronal function. It paved the way for others to follow, like the study by \citealp{KLWH2001}, where they investigated the question of how ITD computation maps can arise ontogenetically in the laminar nucleus of the barn owl. They showed that interaural time differences (ITD) computational maps emerge from the combined effect of a Hebbian spike-based learning rule and its transmission along the presynaptic axon. Thus, from this study, another role of axonal delays can be inferred. They shape network structure when coupled with temporal learning rules. Based on this insight, several studies investigated the combined effect of spike timing-dependent plasticity (STDP), axonal conduction delays and oscillatory activity on recurrent connections in spiking networks. \citealp{KBTG2013} demonstrated the selective potentiation of recurrent connections when the beforementioned computational considerations are taken into account. Also, \citealp{HKTI2016} showed that neural selection for memory formation depends on neural competition. In turn, for neural competition to emerge in recurrent networks, spontaneously induced neural oscillation coupled with STDP and axonal delays are a perquisite. 
+Studying the computational properties of axonal transmissions goes as far back as in {cite:p}`LAJ1948`. In this study, it was shown that with the right network setup, axonal delays can be utilized to transform a temporal cue to a spatial one for sound localization. This study was a leading study in terms of using delays explicitly to explain a neuronal function. It paved the way for others to follow, like the study by {cite:p}`KLWH2001`, where they investigated the question of how ITD computation maps can arise ontogenetically in the laminar nucleus of the barn owl. They showed that interaural time differences (ITD) computational maps emerge from the combined effect of a Hebbian spike-based learning rule and its transmission along the presynaptic axon. Thus, from this study, another role of axonal delays can be inferred. They shape network structure when coupled with temporal learning rules. Based on this insight, several studies investigated the combined effect of spike timing-dependent plasticity (STDP), axonal conduction delays and oscillatory activity on recurrent connections in spiking networks. {cite:p}`KBTG2013` demonstrated the selective potentiation of recurrent connections when the beforementioned computational considerations are taken into account. Also, {cite:p}`HKTI2016` showed that neural selection for memory formation depends on neural competition. In turn, for neural competition to emerge in recurrent networks, spontaneously induced neural oscillation coupled with STDP and axonal delays are a perquisite. 
 
-Coupling conduction delays with STDP seems like a reasonable choice. The sign of the STDP rule depends on the order of post- and pre-synpatic spiking, which axonal delays can effectively reverse. For example, if the presynaptic spikes arrive at the synapse before the backpropagated action potential this would lead a synaptic depression. However, reducing the axonal transmission speed would lead to potentiation. In this line of thought, \citealp{MAVT2017} studied the combined role of delays and STDP on the emergent synaptic structure in neural networks. It was shown that, qualitatively different connectivity patterns arise due to the interplay between axonal and dendritic delays, as the synapse and cell body can have different temporal spike order. 
+Coupling conduction delays with STDP seems like a reasonable choice. The sign of the STDP rule depends on the order of post- and pre-synpatic spiking, which axonal delays can effectively reverse. For example, if the presynaptic spikes arrive at the synapse before the backpropagated action potential this would lead a synaptic depression. However, reducing the axonal transmission speed would lead to potentiation. In this line of thought, {cite:p}`MAVT2017` studied the combined role of delays and STDP on the emergent synaptic structure in neural networks. It was shown that, qualitatively different connectivity patterns arise due to the interplay between axonal and dendritic delays, as the synapse and cell body can have different temporal spike order. 
 
-Aside from their role in modeling cortical functions or shaping a network's synaptic structure, another line of research emerged from the seminal work by \citealp{EMI2006}. They showed that when including conduction delays and spike-timing dependent plasticity (STDP) into their simulation of realistic neural models, polychronous groups of neurons emerge. These groups show time-locked spiking pattern with millisecond precision. Subsequent studies investigated the properties and functions of such neuronal groups. For example, \citealp{BSEI2010} demonstrated the natural emergence of large memory content and working memory when the neuronal model exploits temporal codes. Specifically, short term plasticity can briefly strengthen the synapses of specific polychronous neuronal groups (PNG) resulting in an enchantment in their spontaneous reactivation rates.  In a qualitatively different study, \citealp{EIAS2018} showed that networks that exhibit PNG possess potential capabilities that might solve the dynamic binding problem. These networks respond with stable spatio-temporal spike trains when presented with input images in the form of randomized Poisson spike trains. The functionality of these kind of networks emerged due to the interplay of various factors including: i) random distribution of axonal delays ii) STDP ii) lateral, bottom-up and top-down synaptic connections. 
+Aside from their role in modeling cortical functions or shaping a network's synaptic structure, another line of research emerged from the seminal work by {cite:p}`EMI2006`. They showed that when including conduction delays and spike-timing dependent plasticity (STDP) into their simulation of realistic neural models, polychronous groups of neurons emerge. These groups show time-locked spiking pattern with millisecond precision. Subsequent studies investigated the properties and functions of such neuronal groups. For example, {cite:p}`BSEI2010` demonstrated the natural emergence of large memory content and working memory when the neuronal model exploits temporal codes. Specifically, short term plasticity can briefly strengthen the synapses of specific polychronous neuronal groups (PNG) resulting in an enchantment in their spontaneous reactivation rates.  In a qualitatively different study, {cite:p}`EIAS2018` showed that networks that exhibit PNG possess potential capabilities that might solve the dynamic binding problem. These networks respond with stable spatio-temporal spike trains when presented with input images in the form of randomized Poisson spike trains. The functionality of these kind of networks emerged due to the interplay of various factors including: i) random distribution of axonal delays ii) STDP ii) lateral, bottom-up and top-down synaptic connections. 
 
-Finally, it should be noted that most of the studies that incorporate axonal and/or dendritic delays, include them as a non-learnable parameter. Few studies investigated the possibility of training transmission delays in order to enhance the computational capabilities of spiking neural networks (SNN). \citealp{TM2017} proposed an algorithm that modifies the axonal delays and synaptic efficacy in both supervised and unsupervised approaches.  The learning method used approximates the Expectation-Maximization (EM) algorithm and after training, the network learns to  map spatio-temporal input-output spike patterns. Thus, EM is one way to train SNN that are cast as probabilistic models. Another approach that exploits the massive infrastructure that is laid out the deep learning literature is the work by \citealp{HHM2023}. In this work, delays are represented as 1D convolutions through time, where the kernels include a single per-synapse non-zero weight. The temporal position of these non-zero weights corresponds to the desired delays. The proposed method co-trains weights and delays and is based on the Dilated Convolution
-with Learnable Spacings (DCLS) algorithm [\citealp{ITT2023}].  
+Finally, it should be noted that most of the studies that incorporate axonal and/or dendritic delays, include them as a non-learnable parameter. Few studies investigated the possibility of training transmission delays in order to enhance the computational capabilities of spiking neural networks (SNN). {cite:p}`TM2017` proposed an algorithm that modifies the axonal delays and synaptic efficacy in both supervised and unsupervised approaches.  The learning method used approximates the Expectation-Maximization (EM) algorithm and after training, the network learns to  map spatio-temporal input-output spike patterns. Thus, EM is one way to train SNN that are cast as probabilistic models. Another approach that exploits the massive infrastructure that is laid out the deep learning literature is the work by {cite:p}`HHM2023`. In this work, delays are represented as 1D convolutions through time, where the kernels include a single per-synapse non-zero weight. The temporal position of these non-zero weights corresponds to the desired delays. The proposed method co-trains weights and delays and is based on the Dilated Convolution
+with Learnable Spacings (DCLS) algorithm [{cite:p}`ITT2023`].  
 
 In this work we propose a delay learning algorithm that is simple and efficient. The delay learning is mediated by a differentiable delay layer (DDL). This layer can be inserted between any two layers in an SNN in order to learn the appropriate delay to solve a machine learning task. This DDL is architecture agnostic. Also, the method is designed to learn delays separately from synaptic weights. 
 
 ### Methods
 
-The DDL is, mainly, based on a 1D version of the spatial transformer (STN) network \citealp{JSZK2015}. The STN is a differentiable module that can be added into convolutional neural networks (CNNs) architectures to empower them with the ability to spatially transform feature maps in a differentiable way. This addition leads to CNNs models that are invariant to various spatial transformations like translation, scaling and rotation. Image manipulations are inherently  not differentiable, because pixels are a discrete. However, this problem is overcome by the application of an interpolation  (for example bi-linear) after the spatial transformation. 
+The DDL is, mainly, based on a 1D version of the spatial transformer (STN) network {cite:p}`JSZK2015`. The STN is a differentiable module that can be added into convolutional neural networks (CNNs) architectures to empower them with the ability to spatially transform feature maps in a differentiable way. This addition leads to CNNs models that are invariant to various spatial transformations like translation, scaling and rotation. Image manipulations are inherently  not differentiable, because pixels are a discrete. However, this problem is overcome by the application of an interpolation  (for example bi-linear) after the spatial transformation. 
 
 The DDL is a 1D version of the spatial transformer where the only transformation done is translation. Translation of a spike along the time dimension can be thought of as a translation of a pixel along the spatial coordinates. The general affine transformation matrix for the 2D case takes the form in the following equation:
 
@@ -74,7 +74,7 @@ $$
 (u_{1i}(t) - u_{2i}(t))^2 = u_{1i}(t)^2 -2*u_{1i}(t)*u_{2i}(t) + u_{2i}(t)^2
 $$
 
-The synaptic delay learning method, employing Dilated Convolutions with Learnable Spacings, operates by delaying spike trains through a 1D convolution featuring a single non-zero element, equivalent to the synaptic weight, positioned at the appropriate delay value. A distinctive aspect of this method lies in its utilization of Gaussian interpolation to identify the optimal delay. This approach is crucial because delay values are discrete, making them challenging to learn via backpropagation. However, employing interpolation overcomes this obstacle, facilitating the learning of delays with weights through backpropagation through time in arbitrarily deep SNNs. As we have implemented the method precisely as described in the original paper (with the exception of hyperparameters), we direct the reader to the original paper for a comprehensive understanding\citealp{HHM2023}.
+The synaptic delay learning method, employing Dilated Convolutions with Learnable Spacings, operates by delaying spike trains through a 1D convolution featuring a single non-zero element, equivalent to the synaptic weight, positioned at the appropriate delay value. A distinctive aspect of this method lies in its utilization of Gaussian interpolation to identify the optimal delay. This approach is crucial because delay values are discrete, making them challenging to learn via backpropagation. However, employing interpolation overcomes this obstacle, facilitating the learning of delays with weights through backpropagation through time in arbitrarily deep SNNs. As we have implemented the method precisely as described in the original paper (with the exception of hyperparameters), we direct the reader to the original paper for a comprehensive understanding{cite:p}`HHM2023`.
 
 ### Results and discussion
 

paper/sections/new_inh_model/inhibition_model.md

@@ -73,5 +73,5 @@ Figure 3: Loss of contralateral ITD peak-coding for the four different neurons i
 ```
 
 The achievements of this work are to be considered significant in the investigation of the mechanisms used by the mammalian brainstem to perform sound localization. The computational model developed is a good validation platform for the most recent theories concerning the processing of the two binaural cues, ITDs, and
-ILDs. This model has shown, for pure tones of 100 Hz frequency, the physiological functioning of LSO and MSO. The peak-coding strategy applied for the identification of contralateral angles in each MSO can be considered a refinement of the rate-based localization of sounds happening in the LSO. As described in the [Introduction], this type of redundancy is also justified by the evolutionary history of spatial hearing mechanisms in mammals.
+ILDs. This model has shown, for pure tones of 100 Hz frequency, the physiological functioning of LSO and MSO. The peak-coding strategy applied for the identification of contralateral angles in each MSO can be considered a refinement of the rate-based localization of sounds happening in the LSO. As described in the [](#inhib-intro), this type of redundancy is also justified by the evolutionary history of spatial hearing mechanisms in mammals.
 In contrast to birds, where ITDs have always been used as the main binaural cue for sound localization, early mammals first made exclusive use of ILDs as acoustic cues and only later developed a sensitivity to different ITDs, created in tandem with the development of more dedicated low-frequency hearing. This leads to the possible conclusion that strategies similar to those used for the processing of ILDs have also been readapted for the processing of ITDs and that in the brainstem of modern mammals, these two processes occur in parallel, merging at a higher level, and thus providing a more refined and complete spatial map.