{"id":561,"date":"2024-01-20T21:37:26","date_gmt":"2024-01-20T21:37:26","guid":{"rendered":"https:\/\/sinatootoonian.com\/?p=561"},"modified":"2025-12-28T15:34:06","modified_gmt":"2025-12-28T15:34:06","slug":"differentiating-scalar-functions-of-matrices-in-mathematica-in-22-easy-steps","status":"publish","type":"post","link":"https:\/\/sinatootoonian.com\/index.php\/2024\/01\/20\/differentiating-scalar-functions-of-matrices-in-mathematica-in-22-easy-steps\/","title":{"rendered":"Differentiating scalar functions of matrices in 22 easy steps of Mathematica."},"content":{"rendered":"\n
I frequently need to differentiate loss functions with respect to matrices. I usually do this manually, which can be time-consuming and error-prone. Therefore I wanted to see if I could use Mathematica to compute these symbolic derivatives automatically. Mathematica does not have such functionality built in, but ChatGPT suggested achieving it using pattern matching. <\/p>\n\n\n\n
The loss function we’ll differentiate is $$ L(R) = {1 \\over 2} \\|A^T (J + R)^T (J+R) A – D\\|_F^2 + {\\lambda \\over 2}\\|R\\|_F^2.$$ <\/p>\n\n\n\n
Manual differentiation<\/h3>\n\n\n\n
Let’s first differentiate the loss by hand to see what the right answer is. Letting $H = A^T (J + R)^T (J+R) A – D$, we have $$ \\begin{align}L(R) &= {1 \\over 2} \\tr(H H^T) + {\\lambda \\over 2}\\tr(R R^T)\\\\ dL &= \\tr(dH^T H) + \\lambda \\tr(dR^T R)\\\\ &= \\tr(2 A^T dR^T (J + R)A H) + \\lambda \\tr(dR^T R)\\\\ &= \\tr(2 dR^T (J + R)A H A^T ) + \\lambda \\tr(dR^T R)\\\\ &=\\tr(2 dR^T (J+R)A(A^T (J + R)^T (J+R) A – D)A^T + \\lambda \\tr(dR^T R). \\end{align}$$ Therefore, $$\\boxed{\\nabla_R L= 2 (J+R)(A A^T (J + R)^T (J+R) A A^T -A DA^T) + \\lambda R.}$$ That wasn’t so bad. Let’s now try to get the same thing by applying transformation rules.<\/p>\n\n\n\n
Semi-automatic differentiation<\/h3>\n\n\n\n
There are lots of ways to do this, below is just one, and probably not the most efficient one given that I’m a Mathematica novice.<\/p>\n\n\n
First we’ll apply a symbolic differential.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Next we’ll apply a rule that commutes sums with differentials.<\/p>\n\n\n\n<\/figure>\n\n\n\n
We want to bring the 1\/2 factors out of the differentials, so we create a rule to commute differentials with scalars. The first line of the output below describes the rule, and the second line show the result of its application.<\/p>\n\n\n\n<\/figure>\n\n\n\n
We want to do the same with the lambda at the end, so:<\/p>\n\n\n\n<\/figure>\n\n\n\n
We’re aiming at $\\tr[dR^T (\\dots)]$, so next we commute the trace and the differential.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Our next aim is to turn this into the trace of a sum of differentials of dot product, e.g. $ \\sum_i \\tr[d(A_i B_i \\dots)]$. To do this, we’re going to apply a sequence of rules. The first expresses that $(A + B)^T = A^T + B^T$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
The next distributes the dot product over sums, $A.(B+C) = A.B+ A.C$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
The next expresses $(A^T)^T = A$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
The next commutes transposing with scalar multiplication i.e. $(k A)^T = k A^T$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Finally, the last distributes the differential over sums $d(A+B) = d(A) + dB$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
We will now apply these in the order I found by trial and error to be effective. The ‘\/.’ applies the rule once, ‘\/\/.’ applies it repeatedly until the expression doesn’t change.<\/p>\n\n\n\n<\/figure>\n\n\n\n
The next rule distributes transpose over dot products $(A B)^T = B^T A^T$<\/p>\n\n\n\n<\/figure>\n\n\n\n
The next rule extracts out the annoying minus signs from dot products: $A B (-C)D = -ABCD$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Next we apply Leibniz’s rule to expand out the differentials, $d(AB) = d(A)B + A d(B)$. This produces a very long output which I’ve truncated.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Our next aim is to move all the transposed differentials to be left-most in their terms, and the un-transposed differentials to be right-most, using the circular property of the trace, $\\tr(AB) = \\tr(BA)$. <\/p>\n\n\n\n<\/figure>\n\n\n\n
We next apply a transpose to end up with all terms having transposed differentials at the left-most position. We can then read off their contributions to the gradient as whatever the transposed differentials are being dot-producted against.<\/p>\n\n\n\n<\/figure>\n\n\n\n
We only care about differentials of $R$, so we now zero-out the other terms.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Rather than having one big trace we distribute the trace over sums, $\\tr(A+B) = \\tr(A)+ \\tr(B)$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
We pull out the constant terms by computing trace with scalar multiplication, $\\tr(a X)$ = a \\tr(X)$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
We can now extract the gradient as the sum of the terms being dot-producted against $d(R)^T$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
$D1$ is a diagonal matrix, so $D1^T = D1$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Next we’re going to collect some dot product. First, using $AB + AC = A(B+C)$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Next, using $BA + CA = (B+C)A$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Next, that $k BA + k CA= k( B+ C)A$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Next, that $ p A B + q A C = A (p B + qC)$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Finally, we collect transposes, $A^T + B^T = (A+B)^T$.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Ideally, I’d pull out the factor of 2 as well, but couldn’t quite get that to work. Nevertheless, this is the same equation we derived manually above.<\/p>\n\n\n\n
Summary<\/h3>\n\n\n\n
While we did get the right derivative in the end, the process was very ad hoc and even more laborious than manual differentiation. Having to hard-code the rules was interesting because it made me realise how many of these rules we apply, in problem-specific order, to arrive at the gradient. I suppose that’s part of the fun! This was an interesting exercise, but I think I’ll continue to derive my gradients manually for now.<\/p>\n","protected":false},"excerpt":{"rendered":"
I frequently need to differentiate loss functions with respect to matrices. I usually do this manually, which can be time-consuming and error-prone. Therefore I wanted to see if I could use Mathematica to compute these symbolic derivatives automatically. Mathematica does not have such functionality built in, but ChatGPT suggested achieving it using pattern matching. The […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,152],"tags":[21,20,22],"class_list":["post-561","post","type-post","status-publish","format-standard","hentry","category-blog","category-post","tag-differentiation","tag-mathematica","tag-pattern-matching"],"acf":[],"_links":{"self":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/561","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/comments?post=561"}],"version-history":[{"count":25,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/561\/revisions"}],"predecessor-version":[{"id":612,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/561\/revisions\/612"}],"wp:attachment":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/media?parent=561"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/categories?post=561"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/tags?post=561"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-1024x182.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-1-1024x158.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-2-1024x315.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-3-1024x244.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-4-1024x312.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-5-1024x169.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-6-1024x186.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-7-1024x274.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-8-1024x118.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-9-1024x218.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-10-1024x268.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-11-1024x239.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-12-1024x209.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-13-1024x246.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-14-1024x331.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-15-1024x227.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-16-1024x361.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-17-1024x289.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-18-1024x286.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-19-1024x252.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-20-1024x177.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-21-1024x223.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-22-1024x258.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-23-1024x190.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-24-1024x183.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/sinatootoonian.com\/wp-content\/uploads\/2024\/01\/image-25-1024x165.png\")
<\/figure>\n\n\n\n