There continue to be reasons for software to be slow
There continue to be reasons for software to be slow
Dan Luu recently published a blog post<br>There’s no reason for software to be slow anymore<br>which talks about how various kinds of<br>things are cheaper to do nowadays<br>by virtue of having access to LLMs<br>such as building specialized solutions<br>(e.g. JITs, indexes for search-like problems)<br>as well as workload-specific optimizations.
We’re not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance.
I believe that this statement as written is well-intentioned but incorrect,<br>in much of the same way in which the statement<br>made by Lawson is well-intentioned but incorrect, and in the same way<br>that formal methods advocates arguing that<br>an increasingly larger fractionThe phrase “fraction” is deliberate here. If it were “quantity” instead, it really wouldn’t be a debate.
of software<br>will be formally verified are well-intentioned but incorrect.To be clear, I’m very much in favor of better testing, use of formal methods, as well as performance work! I’ve done work along these lines at multiple jobs, including my current one! It’s just that I don’t agree with these predictions about the future.
In essence, the argument that’s been offered in all of these places<br>goes something like:I’m using $ signs, but you could substitute in “engineer E’s time” if you’d like.
Desirable property X used to cost $A over the budget $B
Pre-LLMs, the reason people didn’t aim for X was it was over budget
Post-LLMs, getting X costs $A/N > 1.
If these premises hold, then people will now spend $A/N for X.
On the face of it, if you’ve personally found LLMs useful at improving property X,<br>the argument seems sensible.You might be thinking “this also assumes that people are rational economic actors with perfect knowledge.” Yes, that’s correct. Most of this post will assume that and show how things can go wrong even with such a strong assumption. In practice, yes people are not rational and don’t have perfect knowledge. I’m ignoring that for this post because there’s already a lot of writing on that topic.
But it only works in practice if the premises hold.
I agree that there are situations where these premises hold.<br>Will certain highly experienced people with deep domain expertise<br>(like the ones cited in Luu’s post)<br>do a bunch more optimizations,<br>or work on teams which ship many more optimizations than before?<br>Yeah, I think that’s definitely going to happen.
However, based on what I’ve seen so far,<br>the situations in which the premises hold<br>are far outweighed by the situations in which they do not hold.
In this post, I’m going to give examples of situations<br>in which I’ve seen these premises not holding.
The tolerance for ‘not X’ goes up
(Or: “the desirability for X goes down”)
One of the differences with the advent of LLMs<br>is that the work you would do synchronously<br>now potentially needs to be done asynchronously,<br>due to latency of agentic loop iterations.
As a concrete example of this, I’ve been working<br>on improving git performance for our monorepo<br>at work recently. If you took the performance numbers<br>we see today on a good day, and you gave them to<br>me from 2022, and told me that people<br>find the same numbers acceptable, I would likely<br>have given you a very skeptical or confused look.
As another example, the latency for LLM-based<br>auto-complete used to be much higher than<br>standard IDE auto-complete when it was introduced.This changed later as Cursor and other editors introduced smaller, specialized models for faster completions.
Around that time,<br>if you saw videos of developers live-coding,<br>you’d notice them having small pauses waiting<br>for the LLM suggestions.<br>But historically, one of the reasons auto-completion was purportedly prized<br>was the “instant” feedback!
This point also applies to things like compilation speed,<br>link times, time to run tests etc. In general, people’s tolerances<br>for synchronous work and asynchronous work are quite different.
If you’re a performance-minded person, it can be hard to accept<br>that people are actually fine with putting up with worse performance in software,<br>especially if you already believe that the performance<br>of said software is “too slow.” It can be doubly-frustrating<br>if the same people are willing to put up with worse performance<br>specifically in exchange for more features,<br>especially if you already believe that the said<br>software is “too bloated.”
The budget was zero from the start
Outside of well-paying tech companies that treat developers well,<br>granting them a fair amount of autonomy,<br>it’s common in many companies for the software function<br>to be perceived as a “cost center” instead of a “profit center”.
There may not even be a CI process – it may be entirely reliant on manual QA.<br>Getting budget approvals...