{"id":7546,"date":"2026-04-21T16:19:04","date_gmt":"2026-04-21T15:19:04","guid":{"rendered":"https:\/\/sinatootoonian.com\/?p=7546"},"modified":"2026-04-21T16:19:05","modified_gmt":"2026-04-21T15:19:05","slug":"relating-gain-and-crowding-in-the-diagonal-model","status":"publish","type":"post","link":"https:\/\/sinatootoonian.com\/index.php\/2026\/04\/21\/relating-gain-and-crowding-in-the-diagonal-model\/","title":{"rendered":"Relating Gain and Crowding in the Diagonal Model"},"content":{"rendered":"\n

When we linearized the diagonal model<\/a> we determined the gains relative to unity as $$ \\bdelta = (\\GG^T \\GG + \\lambda \\II)^{-1} \\GG^T \\rr,$$ where $\\rr$ is the vectorized residual $\\SS – \\XX^T \\XX$. We’d like to not just report these numbers, but explain them. Complexity in the explanation derives from the correlations in the off-diagonal terms of $\\GG^T\\GG$, the overlaps between the representations of each unit. We found in that post that ignoring those overlaps doesn’t give a good fit. So we instead stuck with the formula above, but expressed it in terms if whitening the target covariance, and whitening the filters we’re matching that against. That may ultimately be what we have to do, but it does again make things hard to explain. <\/p>\n\n\n\n

Before we started doing mean-subtraction, we explained the gains the diagonal model assigned to a unit in terms of “alignment”, defined as its representation’s overlap with the target, and crowding, how much its representation aligned with that of the others. Can we do something similar here?<\/p>\n\n\n\n

To see how that would work, let’s rewrite the above equation as $$ (\\GG^T \\GG + \\lambda \\II) \\bdelta = \\GG^T \\rr.$$ In terms of single units, this says $$ \\sum_j (\\GG^T \\GG)_{ij} \\delta_j + \\lambda \\delta_i = \\bg_i^T \\rr \\triangleq a_i,$$ where $a_i$ is the unit’s alignment with the residual. <\/p>\n\n\n\n

Rearranging to isolate $\\delta_i$, we get $$ \\|\\bg_i\\|_2^2 \\delta_i + \\sum_{j \\neq i} \\bg_i^T \\bg_j \\delta_j + \\lambda \\delta_i = a_i.$$ We can then solve $$ \\delta_i = {a_i – b_i \\over \\|\\bg_i\\|_2^2 + \\lambda}, \\quad b_i \\triangleq \\sum_{j \\neq i} \\bg_i^T \\bg_j \\delta_j.$$<\/p>\n\n\n\n

Now $b_i$ looks like a crowding term, measuring overlaps of representations from each term. The problem is that we need the deviations $\\delta_j$, from the other units, and these won’t be available until we actually perform the computation. What we’d like is to give experimentalists a simple rule, just based on the target and input representations, that predicts which units will have large gains. We don’t want this rule to depend on the gains of other units, as those won’t be initially available to the experimenters.<\/p>\n\n\n\n

We got around this before, when we weren’t averaging, and were looking at the raw gains not their deviations, by using a crowding exponent: raising each term to a fixed value less than 1. This was purely heuristic. And since the gains were being regularized to be near 1, even setting this exponent to 1, equivalent to assuming all the other gains were exactly 1, gave good results. We can’t do that here because we’re working with deviations and their regularized target is 0.<\/p>\n\n\n\n

It would be useful if there was a simple empirical relationship that we could exploit to approximate $b_i$ without having to know much about the actual gains. To check, let’s plot $b_i$ against $\\hat b_i \\triangleq \\sum_j \\bg_i^T \\bg_j$:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

A few things to notice:<\/p>\n\n\n\n

    \n
  1. The $b_i$ are all negative. That means that the effect of the overlaps is to boost the gain beyond the individual contributions.<\/li>\n\n\n\n
  2. The relationship is almost perfectly linear.<\/li>\n\n\n\n
  3. The slope of the linearity is negative: the larger the overlaps with the rest of the population, the more the boost.<\/li>\n\n\n\n
  4. The intercept is basically zero, so we basically have a proportionality here.<\/li>\n<\/ol>\n\n\n\n

    Are these trivial consequences of a simple relationship?<\/p>\n\n\n\n

    It’s certainly not a generic property: if I randomly generate $\\GG$ using Gaussian elements, I get:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    We can write our estimate of the residual as $$\\hat \\rr = \\GG \\bdelta = \\sum_i \\bg_i \\delta_i = \\bg_i \\delta_i + \\sum_{j \\neq i} \\bg_j \\delta_j \\triangleq \\bg_i \\delta_i + \\hat \\rr_i,$$ where $\\hat \\rr_i$ is our estimate with the contribution from unit $i$ removed.<\/p>\n\n\n\n

    In these terms, $$ b_i = \\bg_i^T \\sum_{j \\neq i} \\bg_j \\delta_j = \\bg_i^T \\hat \\rr_i.$$ So, it’s the overlap of a unit’s representation with the residual without that unit. If that overlap is large, it will depress the gain on that unit relative to the uncorrelated case. That makes sense – when we assume units are uncorrelated, each should make an independent contribution. If there’s overlap with what the other units can produce, that means the given unit’s contribution wasn’t unique, and can be distributed to the others, reducing its gain.<\/p>\n\n\n\n

    Our estimate of this term is $$ \\hat b_i = \\bg_i^T \\sum_{j \\neq i} \\bg_i = \\bg_i^T \\overline \\bg_i,$$ where we’ve defined $$ \\overline \\bg_i \\triangleq \\sum_{j \\neq i} \\bg_i,$$ the mean population representation without unit i (up to scaling).<\/p>\n\n\n\n

    What we’ve found above is that $$ b_i \\propto -\\hat b_i.$$<\/p>\n\n\n\n

    This says that the more a unit overlaps with the remaining population vector, the larger its effective boost (since $-b_i$ increases). This seems counterintuitive. <\/p>\n\n\n\n

    To better understand this, lets write the estimate in terms of the population mean.<\/p>\n\n\n\n

    \\begin{align*} \\hat \\rr_i = \\sum_{j \\neq i} \\bg_j \\delta_j &= \\sum_{j \\neq i} (\\bg_j – \\overline \\bg_i) \\delta_j + \\overline \\bg_i \\sum_{j \\neq i} \\delta_j. \\\\ b_i = \\bg_i^T \\hat \\rr_i &= \\bg_i^T \\sum_{j \\neq i} (\\bg_j – \\overline \\bg_i) \\delta_j + \\bg_i^T \\overline \\bg_i \\sum_{j \\neq i} \\delta_j \\\\b_i &= \\bg_i^T \\sum_{j \\neq i} (\\bg_j – \\overline \\bg_i) \\delta_j + \\hat b_i \\sum_{j \\neq i} \\delta_j\\\\ &= d_i + c_i \\hat b_i\\end{align*}<\/p>\n\n\n\n

    This reveals a linear relationship between any given $b_i$ and its corresponding $\\hat b_i$. But the constants of proportionality, $c_i$, and the intercepts, $d_i$, vary per unit:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    Interestingly, the intercept scales linearly with $\\hat b_i$,<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    while the coefficients are more or less constant at approximately $\\sum \\delta_i$<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Hmm…<\/p>\n\n\n\n

    $$\\blacksquare$$<\/p>\n","protected":false},"excerpt":{"rendered":"

    When we linearized the diagonal model we determined the gains relative to unity as $$ \\bdelta = (\\GG^T \\GG + \\lambda \\II)^{-1} \\GG^T \\rr,$$ where $\\rr$ is the vectorized residual $\\SS – \\XX^T \\XX$. We’d like to not just report these numbers, but explain them. Complexity in the explanation derives from the correlations in the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,166,148],"tags":[],"class_list":["post-7546","post","type-post","status-publish","format-standard","hentry","category-blog","category-bulb-io","category-research"],"acf":[],"_links":{"self":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/7546","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/comments?post=7546"}],"version-history":[{"count":64,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/7546\/revisions"}],"predecessor-version":[{"id":7701,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/7546\/revisions\/7701"}],"wp:attachment":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/media?parent=7546"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/categories?post=7546"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/tags?post=7546"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}