In this note I’m going to re-explain canonical link functions.<\/p>\n\n\n\n
Suppose that we have a set of input-output observations $(\\xx_1, t_1), \\dots (\\xx_N, t_N).$ We’re interested finding a mapping from the inputs to the outputs. In particular, we’ll consider the form $ y = f(\\ww^T \\xx).$ For a given activation function $f$, we may decide to fit $\\ww$ by computed a squared loss $$ L(\\ww) = \\sum_n {1 \\over 2} (t_n – y_n)^2 $$ and then minimizing the loss down the gradient, $$ \\nabla_\\ww L = \\sum_n (t_n – y_n) f'(\\ww^T\\xx_n) \\xx_n.$$<\/p>\n\n\n\n
But why is the squared loss the right one to use? And how should we pick $f$ and fit $\\ww$? <\/p>\n\n\n\n
We can begin by ignoring the inputs and focusing on the outputs. We want to assign a probability to each observation. It seems reasonable to assume that all of the observations would come from the same family of probability distributions, but which family?<\/p>\n\n\n\n
A fruitful guess is the exponential family. In particular, the probability we assign to each output is $$ p(t|\\eta, s) = h(t\/s) g(\\eta) \\exp(\\eta t\/s)\/s,$$ where we assume the scale hyperparameter $s$ is fixed throughout but that the parameter $\\eta$ can vary with each observation.<\/p>\n\n\n\n
The intuition is that when e.g. modelling the data as Gaussian, $\\eta$ can set the mean of the distribution, and that is in turn determined by the inputs.<\/p>\n\n\n\n
Now suppose we wanted to find the parameter $\\eta$ for a given observation $t$ using maximum likelihood. The log likelihood up to constant terms is $ \\log p(t|\\eta, s) = \\log g(\\eta) + \\eta t\/s + …,$ so $$ {d \\log p \\over d \\eta} = {1 \\over g(\\eta)} g'(\\eta) + {t \\over s}.$$ By virtue of the exponential family, this is equal to \\begin{align*} {d \\log p \\over d \\eta} &= -{1 \\over s} \\mu(\\eta,s) + {t\\over s} = {1 \\over s} (t -\\mu(\\eta,s)),\\end{align*} where $$\\mu(\\eta, s) \\triangleq \\mathbb \\EE(t|\\eta, s).$$<\/p>\n\n\n\n
So if we were able to adjust $\\eta$ independently for every observation, we would set it so that the mean of the distribution determined by $\\eta$ matched the observation.<\/p>\n\n\n\n
The simple form of this update suggests having our output variable $y$ model the mean of the distribution, $$ y(\\eta) = \\mu(\\eta, s) = -s {g'(\\eta) \\over g(\\eta)}.$$ <\/p>\n\n\n\n
Assuming some well-behaved-ness, we can invert this relationship and write $\\eta$ as a function $\\psi$ of $y$, which is in turn determined by the drive $a$ to $f$, which is determined by the weights: $$ \\eta = \\psi(y) = \\psi(f(a)) = \\psi(f(\\ww^T\\xx)).$$<\/p>\n\n\n\n
We now use the chain rule to compute the gradient of likelihood, \\begin{align*} {dL \\over d\\ww} &= {dL \\over d\\eta} {d\\eta \\over dy} {dy \\over da} {da \\over d\\ww} \\\\ &=(t – y)\\psi'(y) f'(a) \\xx \\end{align*}<\/p>\n\n\n\n
This still doesn’t determine $f$: this expression is valid for any (well-behaved) $f$. However, we can appeal to the difficulty of the computation: each weight update would have to be scaled by its own specific factor $$h(y) = f'(a(y)) \\psi'(y) = f'(f^{-1} (y)) \\psi'(y).$$ This is a static function of values available at each synapse, so it’s even neurally plausible, but it is complicated.<\/p>\n\n\n\n
We can get a simpler update if we pick $f$ appropriately. In particular, if we pick $f$ such that $$ \\psi'(y) f'(a)= 1 \\implies \\psi'(y) = [f'(a)]^{-1} = {da \\over df} = {d f^{-1} \\over dy}.$$ <\/p>\n\n\n\n
One solution to this is $$ f^{-1} (y) = \\eta(y) \\implies f \\circ \\eta = \\text{identity}, $$ i.e. $f$ is the inverse of $\\eta$.<\/p>\n\n\n\n
Let’s work this out for the case of logistic regression. Here our observations consist of inputs $\\xx_i$ mapped to binary targets $t_i \\in \\{0,1\\}$.<\/p>\n\n\n\n
Since the observations are binary, it’s natural to model them as coming from a Bernoulli distribution with parameter $\\theta$. So, $$ t_i \\sim p(t_i|\\theta_i) = \\text{Ber}(\\theta_i) = \\begin{cases} 1 & \\quad \\text{with probability } & \\theta_i \\\\ 0 & \\quad \\text{with probability } & 1 – \\theta_i \\end{cases}.$$ Notice that $\\theta_i$ is also the mean. The idea is that each sample $\\xx_i$ sets its own corresponding mean $\\theta_i$, and that in turn determines the binary observation.<\/p>\n\n\n\n
To map this into the exponential family, let’s first drop the index $i$ for clarity. We can write the Bernoulli distribution as \\begin{align*} p(t|\\theta) &= \\text{Ber}(\\theta)\\\\ &= \\theta^t (1 – \\theta)^{1 – t}\\\\ &= \\left({\\theta \\over 1 – \\theta}\\right)^t (1 – \\theta) \\\\ &= e^{t \\ln(\\theta\/1-\\theta)} (1 – \\theta).\\end{align*}<\/p>\n\n\n\n
Comparing this to the form we need for the exponential family, we see that our “natural parameter” $\\eta$ is the log odds: $$ \\eta = \\ln {\\theta \\over 1 – \\theta}.$$ <\/p>\n\n\n\n
The remaining term $(1 – \\theta)$ is our $g(\\eta)$, we just have to express it in terms of $\\eta$. From our expression for $\\eta$ we see that $$ \\theta = {e^\\eta \\over 1 + e^\\eta} \\implies 1 – \\theta = {1 \\over 1 + e^\\eta} = g(\\eta).$$<\/p>\n\n\n\n
We don’t have a scale parameter, so $$y(\\eta) = – {g'(\\eta) \\over g(\\eta)} = {e^\\eta \\over 1 + e^\\eta} = {1 \\over 1 + e^{-\\eta}}.$$ <\/p>\n\n\n\n
So we see that the sigmoidal activation function that we frequently encounter is not arbitrary, but is determined by our probabilistic model of the observations!<\/p>\n\n\n\n
We can also check that it $y(\\eta)$ is indeed the mean parameter: \\begin{align*} y(\\eta) &= {e^\\eta \\over 1 + e^\\eta}\\\\ &= {e^{\\ln(\\theta\/1-\\theta)} \\over 1 + e^{\\ln(\\theta\/1-\\theta)}}\\\\ &= {{\\theta \\over 1 – \\theta} \\over 1 + {\\theta \\over 1 – \\theta}}\\\\ &= {{\\theta \\over 1 – \\theta} \\over {1 \\over 1 -\\theta}}\\\\ &= \\theta,\\end{align*} as expected.<\/p>\n\n\n\n
$$\\blacksquare$$<\/p>\n","protected":false},"excerpt":{"rendered":"
In this note I’m going to re-explain canonical link functions. Suppose that we have a set of input-output observations $(\\xx_1, t_1), \\dots (\\xx_N, t_N).$ We’re interested finding a mapping from the inputs to the outputs. In particular, we’ll consider the form $ y = f(\\ww^T \\xx).$ For a given activation function $f$, we may decide […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-8626","post","type-post","status-publish","format-standard","hentry","category-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/8626","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/comments?post=8626"}],"version-history":[{"count":81,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/8626\/revisions"}],"predecessor-version":[{"id":8709,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/8626\/revisions\/8709"}],"wp:attachment":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/media?parent=8626"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/categories?post=8626"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/tags?post=8626"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}