PyTorch: a reference language — PyTorch DevLogPyTorch: a reference language<br>Edward Z. Yang (@ezyang) ·<br>July 25, 2026<br>· 4 min read<br>compilertorch.compileautogradverificationllm<br>A reference implementation is a simplified but complete version of a system<br>that trades performance in return for clarity. We might then say a reference<br>“language” is the fabric of APIs and conventions from which these<br>implementations are cut. At first glance, PyTorch obviously is a reference<br>language: it is, after all, commonly called the lingua franca of modern deep<br>learning. But upon a closer look, there is confusion:<br>Reference implementations usually aren’t deployed to production. But<br>I do my training jobs with PyTorch!
Everyone’s writing kernels with kernel DSLs. What is the role of PyTorch if<br>it’s just gluing kernels together?
AI coding will eventually mean that any stack can be rewritten from scratch;<br>why does being written in PyTorch matter?
So for me, recently, an unusually clarifying perspective has been to think of<br>PyTorch as playing a dual role: as both the reference language and the<br>implementation language. When the scale is not too large or the compiler is<br>working well, the reference implementation can ship to production. But<br>increasingly, I think it will be more and more natural to think of the<br>reference implementation as a software artifact that stands apart from the<br>actual production implementation, by which we can verify the correctness of<br>the production implementation. One implementation to research in, one<br>implementation to scale with, and one verifier to, in the darkness, bind them.<br>The clearest demonstration of this is in the modern usage of kernel DSLs. The<br>traditional, compiler-maximalist view argues that end users should write<br>implementations of NN modules using a high level API (e.g., a<br>Numpy/PyTorch-style API) which a compiler then determines how to compile into<br>an optimized form. But for the most important operations like matrix<br>multiplies and attention, it is not easy for compilers to guarantee peak<br>performance; the proliferation of kernel DSLs has made it dramatically<br>simpler for people to achieve optimal performance by explicitly spelling out<br>tiling and data movement. Does this eliminate the high level API? Usually<br>not: it’s pretty useful to have a reference implementation in plain PyTorch,<br>and most kernel authors will maintain one in parallel with the optimized<br>kernel, verifying correctness with numerical tests.<br>In the same way kernel DSLs have changed how production implementations of<br>operators can be written, I think coding agents change the way production<br>implementations of train steps can be written. Traditionally, we think of<br>autograd as a core part of PyTorch’s value proposition, because it guarantees<br>you will get correct derivatives. However, at scale, the implicit backwards<br>graph becomes an albatross around one’s neck: the majority of your compute is<br>hidden away, with no opportunity to interact with it with normal debugging<br>tools or apply fusions to it in the same way you can do it in eager forwards<br>code. With a compiler, it is possible to modify the backwards graph with,<br>e.g., a pattern match, but this is brittle and a less nice experience than<br>just swapping a call from a reference implementation to a hand-written kernel.<br>This is not a new observation: the now defunct<br>Tangent library was built on the<br>proposition that source-to-source automatic differentiation could be useful.<br>The new recipe looks like this. Keep the traditional PyTorch<br>autograd-friendly code as the reference implementation. Use LLMs to generate<br>an explicit forward-backward version of the code, which can be optimized<br>separately from the reference implementation. Unlike pattern matching, you<br>never have to worry about your optimizations failing to apply. The cost is<br>that the reference and the real implementation can diverge: we need a verifier<br>that shows us they are equivalent. This verifier can be implemented simply<br>with a bitwise equivalence test, or implemented as some sort of graph capture<br>and structural equivalence, in the tradition of translation validation. To<br>ensure the verifier works when one side has a fusion the other doesn’t, you<br>only need to provide a reference implementation of the fusion (an inverse<br>pattern match, if you will!).<br>I am not going to claim that this recipe is right for everyone. It turns out<br>PyTorch, the reference language, is a pretty good executable spec, and at the<br>end of the day what really matters is how quickly you get the experimental<br>results you need. But, having spent a lot of my time recently thinking about<br>what it means for PyTorch to excel at frontier training–and in particular<br>whether or not it is necessary for PyTorch to disrupt itself as scaling<br>continues–I feel that this perspective helps bridge the old and the new. An<br>open question Horace He posed last year was this: “How can we get all of the<br>control of eager-mode execution...