The Unexamined Model Is Not Worth Training
The Unexamined Model Is Not Worth Training
Tags: academia, research
Published on Sunday, July 12, 2026
" Previous post: Some Comments on “Magnifica …
Just as Socrates said that “The unexamined life is not worth living,”,<br>so the unexamined model is not worth training (in machine learning). We<br>often do it anyway but we really shouldn’t.1
Many machine learning researchers, I myself being no exception, have<br>a soft spot for what one may call unexamined engineering practices.2<br>One of these is our propensity for joining together a bunch of<br>different components, optimistically referring to them as a novel<br>architecture, and then go on training said architecture. If it works—great, it must<br>have been an act of genius invention. It if does not<br>work—duh-doy, it was an obvious mistake anyway. Sometimes, to make<br>it work, we start tinkering with it until it does. We then declare this<br>to be a win, write it up, and move to the next project. But after the<br>excitement settles, we often find that we were wrong about some key<br>assumptions of our model.
For instance, consider a message-passing graph neural network. Its main<br>idea is to take a graph with node attributes and, by exchanging messages<br>with neighboring nodes, create a per-vertex and, ultimately, a per-graph<br>representation of the graph, which may be gainfully used for downstream<br>tasks like classification. There is absolutely nothing intrinsically<br>wrong with this approach; in fact, numerous interesting architectural<br>choices enable adjusting this to a variety of research domains.3
But—there is always a but—a closer look shows that such graph neural<br>networks rely on the graph even when the graph contains spurious<br>information that is not required for solving a task.4 Fair enough, we<br>might say, because that is what we would expect them to do, being<br>graph neural networks. Somewhat paradoxically, message-passing graph neural networks<br>also show themselves to be incapable of leveraging important properties of<br>a graph. For instance, their ability to detect a cycle<br>is roughly determined by how many layers such a network has, or,<br>equivalently, how many rounds of message passing it<br>performs.5 The situation becomes even weirder for some datasets,<br>which, upon further inspection, turn out to be best processed by<br>ignoring the graph structure altogether or, more generally, using<br>a different graph than the input graph.
So, what is going on here? Are we, like the alchemists of yore, mixing<br>things together in the quest for the Philosopher’s stone<br>artificial general intelligence?<br>Partially, yes. We unfortunately tend to not examine our models (and<br>our data—but that’s a different story) as seriously as we should.<br>Instead, we are content if we get the “best” numbers in comparison to<br>all other models. Whether the task that we “solved” is hard or not,<br>whether the other models are well-trained or not, whether our model is<br>actually leveraging some of the problem structure to solve the task or<br>not—all of these things become secondary to the warm glow of “reaching<br>SOTA,” i.e., reaching the fabled “state of the art.”
I sometimes observe myself giving in to this feeling, and I would liken<br>it to the obsession of a somewhat strange butterfly collector. Instead<br>of meticulously trying to collect different species and studying them,<br>the collector only wants “something new” for their collection. There is<br>no rhyme or reason to it, and the collection is not ordered in any way,<br>but yet the collector continues…
If we do not make a conscious effort to address it—sometimes fighting<br>against our very own proclivities and (misplaced) incentives—we end up<br>believing the wrong things about our models. I want to close this post<br>with a particularly surprising case. In recent work, my group and<br>I studied the behavior of neural networks operating on manifolds,<br>i.e., specific topological domains.<br>Much to our surprise, it turns out that, at least for “classical”<br>topological properties like homology groups, there are no discernible<br>improvements in performance between graph neural networks and<br>architectures that perform message passing on higher-order structures<br>like manifolds.
However, what looks like a pretty bad result at first is actually good<br>news in disguise: It shows that there is a new research gap to address;<br>it also indicates that we may be—at least when it comes to these topological<br>properties—focusing on the wrong model paradigm. Without diving<br>deeper into how models process such data, we would have never<br>uncovered this fascinating phenomenon.
This serves as a good lesson for me and I vow to examine things more<br>closely moving forward.
Yes, this is one of those annoying “Do as I say, not as I do”-type of posts. I will try to make bearable, though. ↩︎
I am guessing that actual engineers scoff at us. Or maybe they have given up and consider us...