Lateral excitation: Notes on computational neuroscience.
-
Changing regularization
This morning it occurred to me that the problems we’re having with our equation \begin{align}S^2 Z^2 S^2 – S C S = \lambda (Z^{-1} – I)\label{main}\tag{1}\end{align} are due to the regularizer we use, $\|Z – I\|_F^2$. This regularizer makes the default behavior of the feedforward connections passing the input directly to the output. But it’s…
-
Wrangling quartics, V
Yesterday I went to discuss the problem with one of my colleagues. He had the interesting idea of modelling $S$, and especially $S^2$, as low rank, in particular as $S = s_1 e_1 e_1^T$. That is, shifting the focus on $S$ from $Z$. I tried this out today, and although it didn’t quite pan out,…
-
Wrangling quartics, IV
I’m trying to make some sense of $$ {1 \over \la’} \left(S^2 \wt Z_{UU} S^2 – S \wt C_{VV} S\right) + I = \wt Z_{UU}^{-1}. \label{start}\tag{1}$$ Below I’m going to drop all the tildes and subscripts, for clarity. If we left multiply by $Z$ we get $$ {1 \over \la’} Z(S^2 Z^2 S^2 – S…
-
Wrangling quartics, III
We are trying to understand the connectivity solutions $Z$ found when minimizing the objective $$ {1 \over 2 n^2 } \|X^T Z^T Z X – C\|_F^2 + {\la \over 2 m^2}\|Z – I\|_F^2.$$ Recap We found in the previous post that solutions satisfy$$ {1 \over \la’} \left(S^2 \wt Z_{UU}^2 S^2 – S \wt C_{VV} S…
-
How many neurons or trials to recover signal geometry?
This my transcription of notes on a VVTNS talk by Itamar Landau about recovering the geometry of high-dimensional neural signals corrupted by noise. Caveat emptor: These notes are based on what I remember or hastily wrote down during the presentation, so they likely contain errors and omissions. Motivation The broad question is then: Under what…
-
Wrangling quartics, II
In the last post on this topic, we saw that when optimizing the objective$$ {1 \over 2 n^2 } \|X^T Z^T Z X – C\|_F^2 + {\la \over 2 m^2}\|Z – I\|_F^2,$$ any solution $Z$ is symmetric and satisfies $${2 \over n^2} \left( XX^T Z^2 XX^T – X C X^T\right) + {\la \over m^2} I…
-
Inverting arrowhead matrices.
I need to invert a matrix of the form $$ M = I + S^2 H S^2,$$ where $H$ is a symmetric matrix, and $S^2$ is diagonal. The elements of $S^2$ drop off very quickly, so what remains of $H$ are its first column and first row, scaled by $S_{1}^2 S^2$. The result is that…
-
How many lateral dendrites cross a granule cell arbor?
The projection neurons of the olfactory bulb are the mitral cells and tufted cells. Most mitral cells don’t communicate with each other directly. Instead, they interact through the synapses that their lateral dendrites make onto granule cell abors. Activation of these synapses excites the target granule cells, which in turn inhibit the mitral cells that…
-
Decomposing connectivity
While working on optimizing connectivity for whitening (see below) I remembered that it can be useful to decompose connectivity matrices relating neurons into components relating pseudo-neurons. In this post, I’ll show how this can be done, and highlight its application to the whitening problem. I will assume that our $N \times N$ connectivity matrix $W$…
-
Why mean, median, mode?
Recently while thinking about covariances I got to thinking about why we define an even simpler statistic, the mean, as we do. That’s what this post is about. Suppose we have a dataset $X$ consisting of $N$ numbers $x_1 \dots x_N$. Their mean, $\overline x,$ is of course $$ \overline{x} = {1 \over N} \sum_{i=1}^N…
-
Differentiating scalar functions of matrices in 22 easy steps of Mathematica.
I frequently need to differentiate loss functions with respect to matrices. I usually do this manually, which can be time-consuming and error-prone. Therefore I wanted to see if I could use Mathematica to compute these symbolic derivatives automatically. Mathematica does not have such functionality built in, but ChatGPT suggested achieving it using pattern matching. The…
-
The closest rotation to a scaling
What is the closest rotation to a pure scaling? Intuitively, it should be the null rotation, the identity. One way to see this might be to consider starting with the identity scaling, for which it’s clearly true. If we then scale along one of the coordinate axes, there doesn’t seem to be any ‘torque’ that…
-
Wrangling quartics
The problem we have is to minimize $${1\over 2} \|A ^T Z^T Z A – B \|_F^2 + {\lambda \over 2} \|Z – I\|_F^2,$$ where $B$ is symmetric. This is quartic over $Z$ so its solution will be the roots of some cubic function of $Z$, which is likely intractable. However, this problem might have…
-
Nearest orthonormal matrix
We sometimes need to find the nearest orthonormal matrix $Q$ to a given square matrix $X$. This can easily be done using Lagrange multipliers, as I’ll show here. By ‘nearest’ we will mean in the element-wise sum-of-squares sense. This is equivalent to the squared Frobenius norm, so our optimization problem is $$ \argmin_Q \; {1…
-
Estimating the intrinsic dimensionality of data with the participation ratio
Many datasets are samples of the values of a given set of $N$ features. We can visualise these data as points in an $N$-dimensional space, with each point corresponding to one of the samples. Visualization encourages geometric characterization. A basic geometric property is dimensionality: what is the dimension of the space in which the data…
-
The nearest dataset with a given covariance
When analyzing neural activity I sometimes want to find the nearest dataset to the one I’m working with that also has a desired covariance $\Sigma$. In this post I’m going to show how to compute such a dataset. Let the dataset we’re given be the $m \times n$ matrix $X$, and the new dataset that…
Comments?