{"id":1621,"date":"2024-04-04T10:51:19","date_gmt":"2024-04-04T09:51:19","guid":{"rendered":"https:\/\/sinatootoonian.com\/?p=1621"},"modified":"2025-12-27T16:04:56","modified_gmt":"2025-12-27T16:04:56","slug":"changing-regularization-ii","status":"publish","type":"post","link":"https:\/\/sinatootoonian.com\/index.php\/2024\/04\/04\/changing-regularization-ii\/","title":{"rendered":"Changing regularization, II"},"content":{"rendered":"\n

Today I went back to trying to understand the solution when using the original regularization. While doing so it occurred to me that if I use a slightly different regularization, I can get a closed-form solution for the feedforward connectivity $Z$, and without most (though not all) of the problems I was having in my previous attempt at changing the regularizer<\/a>.<\/p>\n\n\n\n

Before describing the idea, let’s recall the connectivity decomposition. Our loss function is $$L(Z) = {1 \\over 2 n^2} \\|Y^T Y – X^T Z^T Z X \\|_F^2 + {\\lambda \\over 2 m^2}\\|Z – I\\|_F^2.$$ As before, we apply SVD to write this as $$L(Z) = {1 \\over 2 n^2} \\|V_Y S_Y^2 V_Y^T – V_X S_X U_X^T Z^T Z U_X S_X V_X^T\\|_F^2 + {\\lambda \\over 2 m^2}\\|Z – I\\|_F^2.$$ Letting $Q$ be an orthonormal completion of the basis formed by the columns of $U_X$, we can decompose the connectivity as \\begin{align} Z &= U_X U_X^T Z U_X U_X^T + U_X U_X^T Z Q Q^T + Q Q^T Z U_X U_X^T + Q Q^T Z Q Q^T \\\\&= U_X \\wt{Z}_{UU} U_X^T + U_X \\wt{Z}_{UQ} Q^T + Q \\wt{Z}_{Q U} U_X^T + Q \\wt{Z}_{QQ}Q^T.\\end{align} <\/p>\n\n\n\n

We can then write the loss as \\begin{align}L(Z) &= {1 \\over 2 n^2} \\|V_Y S_Y^2 V_Y^T – V_X S_X \\wt Z_{UU}^T \\wt Z_{UU} S_X V_X^T – V_X S_X \\wt Z_{QU}^T \\wt Z_{QU} S_X V_X^T\\|_F^2\\\\ &+ {\\lambda \\over 2 m^2}\\left(\\|\\wt{Z}_{UU} – I\\|_F^2 + \\|\\wt{Z}_{QQ} – I\\|_F^2 + \\|\\wt{Z}_{QU}\\|_F^2 + \\|\\wt{Z}_{UQ}\\|_F^2\\right).\\end{align} <\/p>\n\n\n\n

The first term in the loss is the only thing keeping the regularization from sending $Z$ to $I$, and it only involves $\\wt Z_{UU}$ and $\\wt Z_{QU}$. Therefore, we know that regularization will set the remaining components to their regularization targets, and we can just consider the loss as a function of $\\wt{Z}_{UU}$ and $\\wt{Z}_{QU}$: \\begin{align}L(\\wt{Z}_{UU}, \\wt{Z}_{QU}) &= {1 \\over 2 n^2} \\|V_Y S_Y^2 V_Y^T – V_X S_X (\\wt Z_{UU}^T \\wt Z_{UU} + \\wt Z_{QU}^T \\wt Z_{QU}) S_X V_X^T\\|_F^2\\\\ &+ {\\lambda \\over 2 m^2}\\left(\\|\\wt{Z}_{UU} – I\\|_F^2 + \\|\\wt{Z}_{QU}\\|_F^2\\right).\\end{align} <\/p>\n\n\n\n

Notice how in the first term $\\wt{Z}_{QU}$ shows up next to $S_X$, but in the regularization, it shows up alone. The idea is: since we have some freedom in choosing the regularization, why not regularize $\\wt{Z}_{QU} S_X$ and $\\wt{Z}_{UU} S_X$ instead?<\/mark><\/p>\n\n\n\n

In that case, the loss becomes \\begin{align}L(\\wt{Z}_{UU}, \\wt{Z}_{QU}) &= {1 \\over 2 n^2} \\|V_Y S_Y^2 V_Y^T – V_X S_X (\\wt Z_{UU}^T \\wt Z_{UU} + \\wt Z_{QU}^T \\wt Z_{QU}) S_X V_X^T\\|_F^2\\\\ &+ {\\lambda \\over 2 m^2}\\left(\\|\\wt{Z}_{UU} S_X- I\\|_F^2 + \\|\\wt{Z}_{QU} S_X\\|_F^2\\right).\\end{align} <\/p>\n\n\n\n

It’s then natural to define $$ F_U \\triangleq \\wt{Z}_{UU} S_X, \\quad F_Q \\triangleq \\wt{Z}_{QU}S_X$$ in terms of which the loss is
\\begin{align}L(F_U, F_Q) &= {1 \\over 2 n^2} \\|V_Y S_Y^2 V_Y^T – V_X F_U^T F_U V_X^T- V_X F_Q^T F_Q V_X^T\\|_F^2\\\\& + {\\lambda \\over 2 m^2}\\left(\\|F_{U}- I\\|_F^2 + \\|F_{Q}\\|_F^2\\right).\\end{align} Notice how $F_U$ and $F_Q$ have absorbed $S_X$ in the first term.<\/p>\n\n\n\n

We can simplify further by defining stacking $$ F = \\left[\\begin{matrix}F_U \\\\ F_Q \\end{matrix}\\right], \\quad F_0 =\\left[\\begin{matrix} I \\\\ 0 \\end{matrix}\\right]. $$ We then have the loss in terms of $F$ as \\begin{align}L(F) &= {1 \\over 2 n^2} \\|V_Y S_Y^2 V_Y^T – V_X F^T F V_X^T\\|_F^2 + {\\lambda \\over 2 m^2} \\|F – F_0\\|_F^2 \\end{align}<\/p>\n\n\n\n

Since the Frobenius norm is invariant to rotation, we can move $V_Y$ around in the first term. Defining $R = V_Y^T V_X$, we get, after shifting some constants to the regularizer $$L(F) = {1 \\over 2} \\|S_Y^2 – R F^T F R^T\\|_F^2 + \\la’ F^2, \\quad \\la’ \\triangleq {\\la n^2 \\over 2 m^2}$$ <\/p>\n\n\n\n

Taking derivatives, \\begin{align}\\nabla_F L &= – 2 F (R^T (S_Y^2 – R F^T F R^T)R) + 2\\lambda’ (F – F_0)\\\\
&=- 2 F (R^T S_Y^2 R – F^T F) + 2 \\lambda’ (F – F_0).\\end{align}<\/p>\n\n\n\n

Setting this to zero and left-multiplying by $F^T$, we get
$$ F^T F (R^T S_Y^2 R – F^T F) = \\lambda (F^T F – F^T F_0).$$<\/p>\n\n\n\n

Left-multiplying by $(F^T F)^{-1}$ and rearranging $$ R^T \\left(S_Y^2 – \\lambda’ \\right) R = F^T F- \\lambda’ (F^TF)^{-1} F^T F_0.$$<\/p>\n\n\n\n

Applying SVD $ F= U_F S_F V_F^T,$ and \\begin{align} R^T \\left(S_Y^2 – \\la’ \\right) R &= V_F S_F^2 V_F^T- \\la’ V_F S_F^{-2} V_F^T V_F S_F U_F^T F_0\\\\ &= V_F S_F^2 V_F^T- \\la’ V_F S_F^{-1} U_F^T F_0.\\end{align}<\/p>\n\n\n\n

The solution to this is to set $U_F^T = [V_F^T, 0].$ In that case $U_F^T F_0 = V_F^T$, and \\begin{align} R^T \\left(S_Y^2 – \\la’\\right) R &= V_F \\left(S_F^2 – {\\la’ \\over S_F} \\right) V_F^T,\\end{align} so that $$ \\boxed{V_F = R^T = V_X^T V_Y.}$$ The singular values $S_F$ are then the solutions to the independent cubic equations
$$\\boxed{S_F^3 + \\left(\\la’ – S_Y^2\\right)S_F – \\la’ = 0.}$$ Checking the limits,
$$ \\lim_{\\lambda \\to 0} S_F = S_Y, \\quad \\lim_{\\lambda \\to \\infty} S_F = 1.$$<\/p>\n\n\n\n

So we get $$ \\boxed{F_U = V_F S_F V_F^T, \\quad F_Q = 0}$$ with $V_F$ and $S_F$ as above, from which we get \\begin{align}\\boxed{\\wt{Z}_{UU} = V_F S_F V_F^T S_X^{-1}, \\quad \\wt Z_{QU} = 0.}\\end{align}<\/p>\n\n\n\n

Remark:<\/strong> That was a lot of work to derive that $F_Q = 0$. If we knew that at the beginning we wouldn’t need the derivative of the loss and could just derive the expression for $F_U$ directly from it. I wonder if there’s a more direct way to see that $F_Q$ must be 0?<\/em><\/p>\n\n\n\n

Translating this back to connectivity space,
\\begin{align} Z &= U_X \\wt{Z}_{UU} U_X^T\\\\ &= U_X V_F S_F V_F^T S_X^{-1} U_X^T \\\\ &= U_X V_X^T V_Y S_F V_Y^T V_X S_X^{-1} U_X^T.\\end{align}<\/p>\n\n\n\n

We can interpret this as
$$ Z = \\underbrace{(U_X V_X^T)}_{\\text{Nearest `rotation’ to } X} \\cdot \\underbrace{(V_Y S_F V_Y^T)}_{\\text{Approximately } \\sqrt{Y^T Y}} \\cdot \\underbrace{(V_X S_X^{-1}U_X^T)}_{\\text{Left pseudoinverse of } X}$$<\/p>\n\n\n\n

When applied to $X$, we get
$$ Z X = U_X V_X^T V_Y S_F V_Y^T V_X S_X^{-1} U_X^T U_X S_X V_X^T = U_X V_X^T V_Y S_F V_Y^T .$$<\/p>\n\n\n\n

Finally, $$ X^T Z^T Z X = V_Y S_F V_Y^T V_X U_X^T U_X V_X^T V_Y S_F V_Y^T = V_Y S_F^2 V_Y^T.$$ <\/p>\n\n\n\n

This will tend to $Y^T Y$ at low regularization, and to the identity at high regularization. The latter makes sense in hindsight because our regularization is in the eigenspace of $U_X$, and the connectivity is such that the corresponding pseudounits produce unit variance at the output.<\/p>\n\n\n\n

Numerical verification<\/h3>\n\n\n\n

Here is what an example $F$ matrix looks like for one of the fits:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

So $F_Q$ is indeed zero, and $F_U$ is symmetric. I can then compare $V_F$ and $S_F$ to their predicted values:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This shows a good match modulo the sign-flipping on the eigenvectors, which is unavoidable due to their inherent sign ambiguity.<\/p>\n\n\n\n

So, the calculations seem correct.<\/p>\n\n\n\n

$$\\begin{flalign*} && \\phantom{a} & \\hfill \\square \\end{flalign*}$$<\/p>\n","protected":false},"excerpt":{"rendered":"

Today I went back to trying to understand the solution when using the original regularization. While doing so it occurred to me that if I use a slightly different regularization, I can get a closed-form solution for the feedforward connectivity $Z$, and without most (though not all) of the problems I was having in my […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,148],"tags":[36,52,56,3,50,57,40],"class_list":["post-1621","post","type-post","status-publish","format-standard","hentry","category-blog","category-research","tag-calculation","tag-decorrelation","tag-matrix-factorization","tag-optimization","tag-regularization","tag-whitening","tag-work"],"acf":[],"_links":{"self":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/1621","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/comments?post=1621"}],"version-history":[{"count":137,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/1621\/revisions"}],"predecessor-version":[{"id":1803,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/1621\/revisions\/1803"}],"wp:attachment":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/media?parent=1621"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/categories?post=1621"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/tags?post=1621"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}