In this post I explore one possible unconstrained connectivity solution that turns out to not work.
As before, the loss function we’re optimizing is
$$ L(\ZZ) = {1 \over 2} \|\XX^T \ZZ^T \ZZ \XX – \CC\|_F^2 + {\lambda \over 2 }\|\ZZ – \II\|_F^2.$$
The gradient above is $$ \nabla_\ZZ L = \ZZ (2 \XX \bE \XX^T) + \lambda (\ZZ – \II),$$ where $\bE$ is the term in the first norm above.
Setting this to zero and solving for $\ZZ,$ we get
$$ \ZZ = (2 \XX \bE \XX^T/\lambda + \II)^{-1} .$$
This is symmetric, so can be eigendecomposed as $$\UU_Z \SS_Z^{-1}\UU_Z^T = \beta \XX (\XX^T \UU_Z \SS_Z^2 \UU_Z^T \XX – \CC)\XX^T + \II,$$ where $\beta = 2/\lambda.$ Defining $$\DD \triangleq \beta^{-1} (\SS_Z^{-1} – \II),$$ we get $$\UU_Z \DD \UU_Z^T = \XX (\XX^T \UU_Z \SS_Z^2 \UU_Z^T \XX – \CC)\XX^T.$$ Multiplying on the left by $\UU_Z^T$ and on the right by $\UU_Z$, we get
$$ \DD = \UU_Z^T \XX \XX^T \UU_Z \SS_Z^2 \UU_Z^T \XX \XX^T \UU_Z -\UU_Z^T \XX \CC \XX^T \UU_Z.$$
Defining the $$\QQ \triangleq \UU_Z^T \XX,$$ we can write this as $$ \DD = \QQ \QQ^T \SS_Z^2 \QQ \QQ^T- \QQ \CC \QQ^T.$$
By applying the right eigenvectors of $\CC$ to $\XX$ in the original loss, we can replace $\CC$ by $\SS_Y^2$. So we arrive at an interesting “quadratic” equation involving $\QQ$ and diagonal terms, $$ \DD = \QQ \QQ^T \SS_Z^2 \QQ \QQ^T – \QQ \SS_Y^2 \QQ^T.$$
Note: We arrived at this also in Wrangling Quartics, IV.
Guessing a solution
Now suppose $\QQ \QQ^T = \bLa^2$ is diagonal. Then we get, after rearrangement $$\QQ \CC \QQ^T = \bLa^2 \SS_Z^2 \bLa^2 – \DD.$$ This implies that $\QQ$ diagonalizes $\CC = \VV_C \SS_C^2 \VV_C^T$, so $$\QQ = \bLa \VV_C^T.$$
Assuming square matrices
The situation I have is one where $\QQ$ is a tall matrix, since we have fewer odours than cells. But for now suppose we had as many cell as odours, so all the matrices were square. We have $$ \QQ^T = \XX^T \UU_Z = \VV_C \bLa.$$
In otherwords, $\XX^T$ transforms the columns of $\UU_Z$ into the eigenvectors of $\CC$! Is that the case?
No – the $\QQ \QQ^T$ I find is not diagonal. And maybe it can’t be. That would imply $$ \QQ = \bLa \VV_C^T = \UU_Z^T \XX \implies \XX = \UU_Z \bLa \VV_C^T,$$ which is not the case in general – we can set $\UU_Z$ and $\bLa$, but the right eigenvectors of $\XX$ won’t be $\VV_C$ in general.
$$\blacksquare$$
Leave a Reply