{"id":5408,"date":"2026-03-13T19:36:32","date_gmt":"2026-03-13T19:36:32","guid":{"rendered":"https:\/\/sinatootoonian.com\/?p=5408"},"modified":"2026-03-13T19:36:33","modified_gmt":"2026-03-13T19:36:33","slug":"neurips-2025-day-1","status":"publish","type":"post","link":"https:\/\/sinatootoonian.com\/index.php\/2026\/03\/13\/neurips-2025-day-1\/","title":{"rendered":"Neurips 2025 Day 1"},"content":{"rendered":"\n<p>My notes on Day of NeurIPS 2025. I wanted to wait till they were more complete, but it&#8217;s three months later now and they&#8217;re useful as as!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Explainable AI<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Attribution<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>System output is a function of:\n<ul class=\"wp-block-list\">\n<li>Training data, which makes the system sensitive to<\/li>\n\n\n\n<li>Input features, that are transformed by<\/li>\n\n\n\n<li>Model components to produce an output<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Approaches for attributing outputs to features\n<ul class=\"wp-block-list\">\n<li>Perturbation based approaches\n<ul class=\"wp-block-list\">\n<li>Compare system output with a feature that without\n<ul class=\"wp-block-list\">\n<li>Game-theoretic approach\n<ul class=\"wp-block-list\">\n<li>Compute marginal contribution of units<\/li>\n\n\n\n<li>Combine these into Shapley metrics<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Produces saliency maps<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Gradient based approaches\n<ul class=\"wp-block-list\">\n<li>Partial derivatives of output with respect to a features<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Linear approximations\n<ul class=\"wp-block-list\">\n<li>Approximate model as locally linear: $f(x) \\approx w^T x + b.$\n<ul class=\"wp-block-list\">\n<li>Often enough to replace $x$ with binary indicator of feature presence.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Data attribution\n<ul class=\"wp-block-list\">\n<li>Perturbation based: e.g. Leave One Out: $f(x) &#8211; f^{-j}(x)$\n<ul class=\"wp-block-list\">\n<li>Game theoretic metric: Data Shapley<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Gradient-based\n<ul class=\"wp-block-list\">\n<li>Loss gradient overlap<\/li>\n\n\n\n<li>Influence functions: how much do parameters change without a specific training example: $\\theta_{\\epsilon, x_j} = \\text{argmin}_\\theta {1 \\over n} \\sum+<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Linear approximations\n<ul class=\"wp-block-list\">\n<li>Skip training, directly predict model output with a linear model.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Component attribution\n<ul class=\"wp-block-list\">\n<li>Component: Neuron, subnetwork, etc.<\/li>\n\n\n\n<li>Perturbation based: Causal Mediation Analysis\n<ul class=\"wp-block-list\">\n<li>Output with and without a specific component.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Game Theoretic: Neural Shapley<\/li>\n\n\n\n<li>Causal tracing:\n<ul class=\"wp-block-list\">\n<li>First, get the normal system output, x \\to f(x): What is the Capital of France?<\/li>\n\n\n\n<li>Then, get the output to a perturbed input x&#8217; \\to f(x&#8217;): What is the Capital of ____?<\/li>\n\n\n\n<li>Finally, bring in component activation from unperturbed case: x&#8217; \\to f_{k^*}(x&#8217;).<\/li>\n\n\n\n<li>Perturb the identified component to obtain target behaviour.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Gradient-based component attribution:\n<ul class=\"wp-block-list\">\n<li>$f_{k^*}(x&#8217;) -f(x&#8217;) \\approx \\nabla_{c_k} f(x&#8217;)[c_k(x) &#8211; c_k(x&#8217;)]$?<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Building Inherently Explainable AI systems<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Explainability as communication channel between the model and the human.<\/li>\n\n\n\n<li>Inherently explainable architectures:\n<ul class=\"wp-block-list\">\n<li>Replacing neural network layers to force explicit representations of human-understandable concepts.\n<ul class=\"wp-block-list\">\n<li>Modifying loss functions accordingly<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Does not cause performance loss.<\/li>\n\n\n\n<li>Transformers to Generalized Additive Models\n<ul class=\"wp-block-list\">\n<li>Backpack Language Models<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Inherently explainable training:\n<ul class=\"wp-block-list\">\n<li>Gradient-based and perturbation based approaches can attribute different answers than masking based approaches.<\/li>\n\n\n\n<li>One solution: change training to reflect post-hoc explainaibility paradigm.<\/li>\n\n\n\n<li>E.g. training on randomly masked data.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Concepts and References<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>LogitLens<\/li>\n\n\n\n<li>Shapley \/ Data Shapley \/ Neural Shapley<\/li>\n\n\n\n<li>Causal Mediation Analysis<\/li>\n\n\n\n<li>Kumar et al. 2022, probing classifiers will rely on spurious features.<\/li>\n\n\n\n<li>Backpack Language Models<\/li>\n\n\n\n<li>Generalized Additive Models<\/li>\n\n\n\n<li>Neural Additive Models<\/li>\n\n\n\n<li><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Geometric Deep Learning<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Groups capture symmetries<\/li>\n\n\n\n<li>Invariant neural networks:\n<ul class=\"wp-block-list\">\n<li>Output invariant transformed input is the same: $f(g.x) = f(x)$<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Equivariance:\n<ul class=\"wp-block-list\">\n<li>Output respects transformed input: $f(g.x) = g.f(x)$<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Why do we care? \n<ul class=\"wp-block-list\">\n<li>Learning efficiency<\/li>\n\n\n\n<li>Noise robustness<\/li>\n\n\n\n<li>Can affect loss-landscapes, learning<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Achieving equivariance<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Modifying off-the-shelf (non-equivariant) networks\n<ul class=\"wp-block-list\">\n<li>Canonicalization\n<ul class=\"wp-block-list\">\n<li>E.g. alignment, registration<\/li>\n\n\n\n<li>Problem: not always continuous: small input changes produce large output changes.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Group averaging\n<ul class=\"wp-block-list\">\n<li>Average output of all group transformed inputs<\/li>\n\n\n\n<li>Nice mathematical properties, including continuity<\/li>\n\n\n\n<li>Problem: Groups can be huge,  enumerating all group elements can be hard.\n<ul class=\"wp-block-list\">\n<li>Solution: Don&#8217;t need to use full group, group generators suffice.\n<ul class=\"wp-block-list\">\n<li>Problem: Group generators can be hard to find.\n<ul class=\"wp-block-list\">\n<li>Solution: A not too large random subset of group elements can approximately capture the symmetries.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Building equivariant-networks\n<ul class=\"wp-block-list\">\n<li>Data augmentation: train on group transformed data.\n<ul class=\"wp-block-list\">\n<li>Problem: No guarantee to exactly capture the symmetry.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Weight-sharing: symmetries are reflected in fixed weigh structure.\n<ul class=\"wp-block-list\">\n<li>E.g. CNNs capturing translation invariance.<\/li>\n\n\n\n<li>Problem: Might not be sufficiently expressive.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>From invariant theory:\n<ul class=\"wp-block-list\">\n<li>Build networks sensitive to specific small order polynomial functions of the input.\n<ul class=\"wp-block-list\">\n<li>E.g. to make invariant to rotations, use dot products.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Concepts and References<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Equivariance<\/li>\n\n\n\n<li>SignNet<\/li>\n\n\n\n<li>DeepSet<\/li>\n\n\n\n<li>Villar &#8220;Machine Learning and Invariant Theory&#8221;<\/li>\n\n\n\n<li>&#8220;Symmetries in Neural Network Parameter Space.&#8221;<\/li>\n\n\n\n<li>Weight space learning &#8220;Neural Nets as Data&#8221;<\/li>\n\n\n\n<li>Model merging<\/li>\n\n\n\n<li>Symmetry-invariant optimization <\/li>\n\n\n\n<li>Neural Radiance Fields<\/li>\n\n\n\n<li>Implicit Neural Representations<\/li>\n\n\n\n<li>Any-dimensional models<\/li>\n\n\n\n<li>&#8220;Representational Stability&#8221; B Farb ICM 2014<\/li>\n\n\n\n<li>Manifold hypothesis helps curse of dimensionality if curvature is bounded positive below\n<ul class=\"wp-block-list\">\n<li>E.g. avoid hyperbolas, space-filling shapes.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>My notes on Day of NeurIPS 2025. I wanted to wait till they were more complete, but it&#8217;s three months later now and they&#8217;re useful as as!<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,167],"tags":[],"class_list":["post-5408","post","type-post","status-publish","format-standard","hentry","category-blog","category-conference"],"acf":[],"_links":{"self":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/5408","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/comments?post=5408"}],"version-history":[{"count":3,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/5408\/revisions"}],"predecessor-version":[{"id":7310,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/posts\/5408\/revisions\/7310"}],"wp:attachment":[{"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/media?parent=5408"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/categories?post=5408"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sinatootoonian.com\/index.php\/wp-json\/wp\/v2\/tags?post=5408"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}