When I think of independence my natural tendency is to consider it as a relation between two items, a binary relation. For example, a node of interest $a$ is independent of $b$ when $$p(a,b) = p(a) p(b).$$ But this expression is symmetric in $a$ and $b$. We could have equivalently said $b$ is independent of $a$, even though our node of interest is $a$.
However, when learning about Markov blankets, an equivalent but different ‘unary’ perspective was more useful. In that setting, we want to determine the minimal set, called the Markov boundary, that makes a node conditionally independent of all others.
My initial approach was to consider my node of interest $a$ and compare it to some node $b$ outside the conditioning set $C$ using the binary form of independence above. So, to show that $$p(a,b|C) = p(a|C) p(b|C).$$
This approach put my node of interest and the the comparison nodes on equal footing, even though I wasn’t really interested in the latter.
Reading Section 8.2.2 of Bishop, it turned out to be much easier to use a different, ‘unary’ expression for independence. It’s a reformulation of the above that says $$ p(a|b) = {p(a,b) \over p(b)} = p(a).$$
This is an equivalent expression to the first one, but is ‘unary’ in that the focus remains on $a$ as the variable of interest, with $b$ given a distinct role as the conditioning set.
This ‘unary’ notion was useful because it kept the focus mostly on my node of interest $a$, and the conditioning set. We expressed conditional independence as $$ p(a| \text{all other nodes}) = p(a|C).$$ This expression was also more appropriate in that it didn’t single out a particular node to check independence against, since no such nodes should be distinguished.
The ‘binary’ definitions seem like an undirected notion of independence, where the two nodes are considered symmetrically in a joint distribution, while the ‘unary’ definition is a directed notion, where one node conditions the other.
Anyway, I think it’s interesting to note that these two mathematically equivalent expressions seem to have different semantic content.
$$\blacksquare$$
Leave a Reply