Theories of Deep Learning - by astle dsa
Theoretical Limits
SubscribeSign in
Theories of Deep Learning<br>A high-level view of various theoretical frameworks in deep learning
astle dsa<br>Jul 11, 2026
Share
This field has been blessed with exponential empirical success in the form of architectures and algorithms that simply worked through scaling, while the theory lagged behind1. Although in the past few years, the “gap“ seems to be diminishing, and we are getting multiple theories for different aspects of deep learning. This essay would be a simple high-level overview of all the theories I’ve come across.<br>NOTE : These are mathematically dense frameworks which either provide a language for formalizing the field itself or aim to explain the observed phenomena. This essay is my personal understanding of the theories, after a lot of back and forth with GPT. Hence, kindly take everything with a grain of salt, and kindly mention any mistakes and misunderstandings in the comments !<br>NOTE : I tend to use the words “theory“ and “framework“ inter-changeably in this essay. If that seems annoying or unnecessary, or is mis-leading in some manner, please DM or comment as well !<br>I.
Before we dive into the actual theories and frameworks, I’d like to sketch out the different “sub-domains” in deep learning here, since we shall see that each sub-domain has a different framework which are largely independent of each other2. We have, broadly :<br>Architecture theory : Or the theoretical foundations which includes all the model architectures, and provides for a common language for us to express popular architectures like transformers, RNNs and CNNs as specific instances of a more general framework.
Optimization theory : as the name suggests, theories here try to explain and predict how different optimizers behave based on model architectures and data distributions, in order to find more efficient and better optimizers like Adam, AdamW or Muon.
Functional Theory : theories in this domain focus on the neural network as a whole simulating a function, and try to understand it’s behaviours from that perspective. Frameworks here aim to explain how and why models generalize and understand phenomena like grokking/double descent.
These aren’t formal categories of the field of deep learning per se, as one cannot be isolated from the other, but the papers and their authors have dealt with only one single aspect from above at a time3, hence I’ve mentioned them here.<br>II.
Let us being with,<br>Categorical Deep Learning for architecture theory.<br>Paper : Position, Categorical Deep Learning is an Algebraic Theory of All Architectures<br>Author(s) : Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, Petar Veličković<br>I originally came across this paper while exploring the generalizations of deep learning through Geometric Deep Learning, to then Topological Deep Learning4 before learning about the Position paper. While the first two fields focus on generalizing the data distributions with non-trivial internal structures by introducing certain constraints and modifications to model architectures, in the CDL paper, the authors provide a general framework for describing all model architectures and building a bridge between this theory and implementation through categorical concepts, which can also be implemented in functional programming concepts. In their own words (emphasis added by me),<br>… lack of a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building such a bridge, we propose to apply category theory—precisely, the universal algebra of monads valued in a 2-category of parametric maps —as a single theory elegantly subsuming both of these flavours of neural network design.
As I mentioned earlier, the authors focus on generalizing all neural network architectures, by utilizing tools from category theory5 in order to cover all the operations seen in contemporary deep learning research, while focusing on implementations through grounding it in functional programming. (I will be exploring the implementations in upcoming posts!).<br>In brief: the authors utilize the compositionality of monads to represent many things, like sequential computation or recurrence. A 1-category has morphisms or transformations between two objects (mathematical objects, like vector spaces) and a 2-category has morphisms over morphisms, which can help represent high-order operations like parameter-sharing, different variations of the same operation or operations like automatic differentiation. Hence, the authors propose an algebra of monads in a 2-category space as the mathematical framework for deep learning.
Modular Duality for Optimization theory.<br>Paper : Modular Duality in Deep Learning<br>Author(s) : Jeremy Bernstein, Laker Newhouse<br>The optimization of deep learning models was generally done by using a standard algorithm like stochastic gradient descent or the Adam...