The Map | tensor
Table of Contents The fallThe territoryThe twelve ideasHow this series drawsHow to read thisWhat you can now sayTry it yourselfPick a doorReferences<br>Figure 1. the program this whole series is about.You have typed something like this a thousand times. This series exists so that, by its end, you know everything these lines do. All of it: the Python they touch, the C++ they land in, the graph they record, the kernels they choose, the memory they use, and the two clocks they run on. Each of those words gets a plain meaning on its floor below.This is Part 0, the map. First we go down through all the layers once, fast. Then we draw the territory. Then twelve ideas that make the rest of the codebase predictable. Then how this series works, and how to read it. Nothing here gets its full story. Everything here gets a place, and every full story has a numbered part waiting for it.One promise before we start. Every measured number in this series comes from a small script you can run yourself, linked right where the number appears. I measured these on an Apple M3 Max laptop with torch 2.11.0 [1]. Your numbers will differ. The pattern they make will not.The fall<br>PyTorch is deep. Between your keyboard and the chip there are eight levels. I will call them floors, and this meter shows all of them. It returns through the whole series, so you always know how deep you are.<br>Figure 2. the depth meter. the orange dot marks where you are.The fastest way to learn a building is to go down through it once without stopping. That is this section.Floor one: python<br>floor 2 of 8pythonfull story: Part 5, The Machinery<br>torch.randn looks like a Python function. Ask Python what it actually is:>>> type(torch.randn)<br>'builtin_function_or_method'>Python gives that type only to functions written in compiled code. Compiled code means: code that was translated to machine instructions before you ever installed it, so there is no Python body inside it to read, and no line for your debugger to stop on.So where do those machine instructions live? In shared libraries. A shared library is a file of compiled code that a program loads while it runs. They sit inside the torch package on your disk, and you can look at them (proof):torch._C -> _C.cpython-312-darwin.so (49 KB, the loader)<br>libtorch_cpu.dylib 206.5 MB (tensors and kernels)<br>libtorch_python.dylib 28.5 MB (the python side of the border) p0_the_library.py the proof, ready to read or run """Proof: where the compiled part of pytorch actually lives.
torch._C is a thin compiled stub; the weight of the framework is in<br>the shared libraries next to it. Prints the files and their sizes.<br>"""<br>import glob<br>import os<br>import torch
stub = torch._C.__file__<br>print(f"torch {torch.__version__}")<br>print(f"torch._C -> {os.path.basename(stub)} "<br>f"({os.path.getsize(stub)/1024:.0f} KB stub)")<br>libdir = os.path.join(os.path.dirname(stub), "lib")<br>for lib in ["libtorch_cpu.dylib", "libtorch_python.dylib"]:<br>p = os.path.join(libdir, lib)<br>if os.path.exists(p):<br>print(f"{lib:24s} {os.path.getsize(p)/1024/1024:6.1f} MB")<br>download and run it<br>Read the sizes, and then look at them:<br>Figure 3. drawn to scale by file size. the part of pytorch that python can see is the orange dot.The part of PyTorch you can see from Python is a 49 KB file whose only job is to load the other two. The real body is 235 MB of compiled code. import torch brings it into your process, and after that, calling torch.randn means jumping into that body. Today we only need to know these files exist.This is the first honest surprise of the codebase: the Python you write all day is the smallest layer of it.The boundary<br>floor 3 of 8the boundaryfull story: Part 5, The Machinery<br>The call leaves Python at once. Where does it land?In a C++ function named THPVariable_randn, inside that 28.5 MB library from the last floor. And here is a strange fact you can keep: this function does not exist in the PyTorch repository. Clone the repo, search for the name, and you find nothing. A program writes this function during the build, together with thousands of its siblings. Idea 4 below explains why, and Part 5 shows the program that does the writing.<br>Figure 4. the border between the two languages. every tensor operation crosses it.Crossing this border costs time. To see the cost alone, time the smallest possible operation, where almost no arithmetic hides it (proof):add, 1 element : 0.538 microseconds per call<br>add, 4M elements : 337.264 microseconds per call p2_dispatch_cost.py the proof, ready to read or run """Proof: the fixed cost of one eager op, and why size hides it.
Times the same `a + b` at two sizes. The one-element add is nearly<br>pure machinery (dispatch, wrapping, allocation); the 4M-element add is<br>nearly pure arithmetic. CPU, single process.<br>"""<br>import time<br>import torch
def per_op_us(a, b, iters):<br># warmup<br>for _ in range(2000):<br>a + b<br>t0 = time.perf_counter()<br>for _ in range(iters):<br>a + b<br>return (time.perf_counter() - t0) / iters * 1e6
tiny =...