In this note I’m going to re-explain canonical link functions.
Suppose that we have a set of input-output observations $(\xx_1, t_1), \dots (\xx_N, t_N).$ We’re interested finding a mapping from the inputs to the outputs. In particular, we’ll consider the form $ y = f(\ww^T \xx).$ For a given activation function $f$, we may decide to fit $\ww$ by computed a squared loss $$ L(\ww) = \sum_n {1 \over 2} (t_n – y_n)^2 $$ and then minimizing the loss down the gradient, $$ \nabla_\ww L = \sum_n (t_n – y_n) f'(\ww^T\xx_n) \xx_n.$$
But why is the squared loss the right one to use? And how should we pick $f$ and fit $\ww$?
We can begin by ignoring the inputs and focusing on the outputs. We want to assign a probability to each observation. It seems reasonable to assume that all of the observations would come from the same family of probability distributions, but which family?
A fruitful guess is the exponential family. In particular, the probability we assign to each output is $$ p(t|\eta, s) = h(t/s) g(\eta) \exp(\eta t/s)/s,$$ where we assume the scale hyperparameter $s$ is fixed throughout but that the parameter $\eta$ can vary with each observation.
The intuition is that when e.g. modelling the data as Gaussian, $\eta$ can set the mean of the distribution, and that is in turn determined by the inputs.
Now suppose we wanted to find the parameter $\eta$ for a given observation $t$ using maximum likelihood. The log likelihood up to constant terms is $ \log p(t|\eta, s) = \log g(\eta) + \eta t/s + …,$ so $$ {d \log p \over d \eta} = {1 \over g(\eta)} g'(\eta) + {t \over s}.$$ By virtue of the exponential family, this is equal to \begin{align*} {d \log p \over d \eta} &= -{1 \over s} \mu(\eta,s) + {t\over s} = {1 \over s} (t -\mu(\eta,s)),\end{align*} where $$\mu(\eta, s) \triangleq \mathbb \EE(t|\eta, s).$$
So if we were able to adjust $\eta$ independently for every observation, we would set it so that the mean of the distribution determined by $\eta$ matched the observation.
The simple form of this update suggests having our output variable $y$ model the mean of the distribution, $$ y(\eta) = \mu(\eta, s) = -s {g'(\eta) \over g(\eta)}.$$
Assuming some well-behaved-ness, we can invert this relationship and write $\eta$ as a function $\psi$ of $y$, which is in turn determined by the drive $a$ to $f$, which is determined by the weights: $$ \eta = \psi(y) = \psi(f(a)) = \psi(f(\ww^T\xx)).$$
We now use the chain rule to compute the gradient of likelihood, \begin{align*} {dL \over d\ww} &= {dL \over d\eta} {d\eta \over dy} {dy \over da} {da \over d\ww} \\ &=(t – y)\psi'(y) f'(a) \xx \end{align*}
This still doesn’t determine $f$: this expression is valid for any (well-behaved) $f$. However, we can appeal to the difficulty of the computation: each weight update would have to be scaled by its own specific factor $$h(y) = f'(a(y)) \psi'(y) = f'(f^{-1} (y)) \psi'(y).$$ This is a static function of values available at each synapse, so it’s even neurally plausible, but it is complicated.
We can get a simpler update if we pick $f$ appropriately. In particular, if we pick $f$ such that $$ \psi'(y) f'(a)= 1 \implies \psi'(y) = [f'(a)]^{-1} = {da \over df} = {d f^{-1} \over dy}.$$
One solution to this is $$ f^{-1} (y) = \eta(y) \implies f \circ \eta = \text{identity}, $$ i.e. $f$ is the inverse of $\eta$.
Example: Logistic Regression
Let’s work this out for the case of logistic regression. Here our observations consist of inputs $\xx_i$ mapped to binary targets $t_i \in \{0,1\}$.
Since the observations are binary, it’s natural to model them as coming from a Bernoulli distribution with parameter $\theta$. So, $$ t_i \sim p(t_i|\theta_i) = \text{Ber}(\theta_i) = \begin{cases} 1 & \quad \text{with probability } & \theta_i \\ 0 & \quad \text{with probability } & 1 – \theta_i \end{cases}.$$ Notice that $\theta_i$ is also the mean. The idea is that each sample $\xx_i$ sets its own corresponding mean $\theta_i$, and that in turn determines the binary observation.
To map this into the exponential family, let’s first drop the index $i$ for clarity. We can write the Bernoulli distribution as \begin{align*} p(t|\theta) &= \text{Ber}(\theta)\\ &= \theta^t (1 – \theta)^{1 – t}\\ &= \left({\theta \over 1 – \theta}\right)^t (1 – \theta) \\ &= e^{t \ln(\theta/1-\theta)} (1 – \theta).\end{align*}
Comparing this to the form we need for the exponential family, we see that our “natural parameter” $\eta$ is the log odds: $$ \eta = \ln {\theta \over 1 – \theta}.$$
The remaining term $(1 – \theta)$ is our $g(\eta)$, we just have to express it in terms of $\eta$. From our expression for $\eta$ we see that $$ \theta = {e^\eta \over 1 + e^\eta} \implies 1 – \theta = {1 \over 1 + e^\eta} = g(\eta).$$
We don’t have a scale parameter, so $$y(\eta) = – {g'(\eta) \over g(\eta)} = {e^\eta \over 1 + e^\eta} = {1 \over 1 + e^{-\eta}}.$$
So we see that the sigmoidal activation function that we frequently encounter is not arbitrary, but is determined by our probabilistic model of the observations!
We can also check that it $y(\eta)$ is indeed the mean parameter: \begin{align*} y(\eta) &= {e^\eta \over 1 + e^\eta}\\ &= {e^{\ln(\theta/1-\theta)} \over 1 + e^{\ln(\theta/1-\theta)}}\\ &= {{\theta \over 1 – \theta} \over 1 + {\theta \over 1 – \theta}}\\ &= {{\theta \over 1 – \theta} \over {1 \over 1 -\theta}}\\ &= \theta,\end{align*} as expected.
$$\blacksquare$$
Leave a Reply