The second is $$ \\XX^T \\JJ [\\bdelta] \\XX = \\sum_i \\delta_i \\widetilde \\xx_i \\xx_i^T.$$ <\/p>\n\n\n\n
We can combine these into a single term $$ \\sum_i \\GG_i \\delta_i, \\quad \\GG_i \\triangleq \\xx_i \\widetilde \\xx_i^T + \\widetilde \\xx_i \\xx_i^T.$$<\/p>\n\n\n\n
We can combine the last term with the target covariance using $$ \\SS – \\XX^T \\JJ \\XX \\triangleq \\bE_0,$$ where the subscript reminds us that this is the error at $\\bdelta = 0.$ <\/p>\n\n\n\n
If we vectorize, forming $$\\GG = [\\bg_1 \\dots \\bg_N], \\quad \\bg_i = \\text{vec}(\\GG_i),$$ and similarly for the other terms, our loss will be $$ L(\\bdelta) = {1 \\over 2} \\|\\GG \\bdelta – \\ee_0\\|_2^2 + {\\lambda \\over 2} \\|\\bdelta\\|_2^2.$$ The gradient is then $$ \\nabla L = \\GG^T (\\GG \\bdelta – \\ee_0) + \\lambda \\bdelta,$$ so the solution is $$ \\boxed{\\bdelta = (\\GG^T \\GG +\\lambda \\II) ^{-1} \\GG^T \\ee_0.}$$<\/p>\n\n\n\n
If we ignore correlations among the $\\bg_i$, this is $$ \\delta_i \\approx {\\bg_i^T \\ee_0 \\over \\|\\bg_i\\|_2^2 + \\lambda}.$$<\/p>\n\n\n\n The linear update is good (orange)! However, the correlations among the columns of $\\GG$ are important, since ignoring them, and just taking the variances, clearly produces a poor estimate.<\/p>\n\n\n\n Now, how to intuitively explain the linear approximation? <\/p>\n\n\n\n One idea I had was to see whether there was an equivalent dataset, one that would give the same coefficients, for the the same $\\ee_0$, but would have decorrelated representational atoms $\\bg_i$, so that the explanation that ignored those correlations would work, and we could explain the results that way. <\/p>\n\n\n\n However, no such dataset exists. Let $\\GG = \\UU \\SS \\VV^T$. Then $$ (\\GG^T \\GG + \\lambda \\II)^{-1} \\GG^T = \\VV {\\SS \\over \\SS^2 + \\lambda \\II} \\UU^T.$$ There are no degrees of freedom here, that would allow us to e.g. rotate $\\VV$ – once $\\GG$ is known, everything is determined.<\/p>\n\n\n\n Another possibility is to switch coordinates: multiply both sides of the above on the left by $\\VV^T$, and we get $$ \\UU^T \\ee_0 = \\left[{\\SS \\over \\SS^2 + \\lambda \\II}\\right]^{-1} \\VV^T \\bdelta.$$ This has the disadvantage that we’re no longer explaining the gains $\\bdelta$ themselves, but some rotated version.<\/p>\n\n\n\n A further possibility is to simply define the raw overlaps as $\\GG^T \\ee_0$, and take the rows of $(\\GG^T \\GG + \\lambda \\II)^{-1}$ as the cell-specific filters.<\/p>\n\n\n\n ChatGPT suggested an improvement on this, where we split the filters into two whitening steps $$ (\\GG^T \\GG + \\lambda \\II)^{-1} \\triangleq \\WW^2, \\quad \\WW = (\\GG^T \\GG + \\lambda \\II)^{-{1 \\over 2}}.$$ We can then write $$ \\delta_i = \\ee_i^T \\WW \\WW (\\GG^T \\ee_0) = \\langle \\WW \\ee_i, \\WW (\\GG^T \\ee_0) \\rangle.$$ <\/p>\n\n\n\n If there were no correlations, $\\delta_i \\propto \\ee_i^T \\GG^T \\ee_0.$ With the correlations, we have to whiten both the filter $\\ee_i$, and the overlaps, $\\GG^T \\ee_0,$ which is what we have above. <\/p>\n\n\n\n I think that’s the best we can do. <\/p>\n\n\n\n $$ \\blacksquare$$<\/p>\n","protected":false},"excerpt":{"rendered":" We’re after insight, not an exact solution. ChatGPT had a good suggestion to linearize the loss around $\\zz = 1$. In this post we do that.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,148],"tags":[161,162,163],"class_list":["post-7014","post","type-post","status-publish","format-standard","hentry","category-blog","category-research","tag-diagonal-model","tag-input-output-transform","tag-iopaper"],"acf":[],"_links":{"self":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/7014","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/comments?post=7014"}],"version-history":[{"count":55,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/7014\/revisions"}],"predecessor-version":[{"id":7543,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/7014\/revisions\/7543"}],"wp:attachment":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/media?parent=7014"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/categories?post=7014"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/tags?post=7014"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2026\/03\/image-9-1024x273.png\")
<\/figure>\n\n\n\nExplanations<\/h2>\n\n\n\n