RL economics, morally charged terms, and “distillation” – Thomas Dullien / Halvar Flake

After a number of Twitter discussions, and repeating myself a lot in these discussions, it is time to write a short note on the economics of advancing LLM capabilities through RL, about principles of propaganda and coining new words, and about my stubborn refusal to use the term “distillation” except in a specific narrow sense.

How do models advance when human-curated data has run out?

It’s been a while since we ran out of human data to train LLMs on. We are training on copies of the internet, large piles of (originally pirated, then purchased-and-scanned-and-wholesale ingested) books, and whatever other data sources we can obtain. This leads to a certain performance plateau, as we haven’t quite figured out how to make the models more data-efficient in training.

The advancements we have seen in coding and mathematics in the last year are mostly due to reinforcement learning. At the highest level, you pose a problem to an LLM that the LLM has a small but nontrivial chance of solving. You then run N copies of the LLM to generate solutions, and you get a small number of solutions and many failures. You can then use the successful solutions as new data to improve your model - moving the weights in a way that helps the model succeed with greater probability.

This is very elegant in a way, because you are kinda pulling yourself up by your own bootstraps. The cost is computational - if you have a 1% chance of finding a solution given your current LLM and current training data, you need to do 100s or 1000s of rollouts to get a reasonable variety of useful solutions.

Once you have a model that can generate a good solution for this problem with high probability, and you make that model available to others, you also provide a much cheaper way of producing the better training data: Third parties can now just ask your model to generate good solutions for them.

So for the second-mover that gets to use your model, improving their model from your model outputs is cheaper, as they can skip the more-or-less-random-search into a high-dimensional solution space and be guided better.

This is a fundamental part of the “closed LLM as a service” business, and it is painful for the leader of the pack because they need to spend money to advance, and others can catch up more cheaply.

Terms of service, copyright law, crimes vs. contract disputes

Copyright law imposes concrete ownership rights on copyrighted material. Pirating material and commercially exploiting it is often a crime.

The frontier labs have all argued that training on public data does not require them to obtain licenses from the copyright holders (a self-serving and somewhat dubious claim). The Llama release further muddied the waters by adding a license to the redistribution of model weights - by law, the output of an algorithm itself (such as model weights) are not a copyrightable object, and Meta just pretended they were. Other model labs followed suit, in the hope of establishing a practical precedent that can then be used to shape legislation in the future.

But a priori, model weights are not copyrightable.

There is an argument, though, that prompts, and the resulting output from the model are copyrightable to the person submitting the prompts. Certainly not to the model provider: Running an algorithm on somebody else’s copyrightable work without human input does not make you the owner of the work. There is no human creativity input, which is the minimum threshold for establishing copyright in our current legal system.

Model providers have no rights to the output of their models if they provide access to these models to third parties.

What rights do model providers have? They have the right to set terms-of-service for their service - e.g. if you don’t use the tool in a way we like, we revoke access to the tool.

Terms-of-service are very different from copyright law - they are essentially private law contracts about the exchange of services between entities. So if a model provider says “you may not use this service to generate training data for your competing LLM”, they can say so, and they have the right to terminate your account if they catch you doing so.

That said - let’s say I was to run a benchmarking service that tests the progress of LLMs against my favorite programming problems, and all I do is (a) run rollouts against these services (b) score the results (c) archive the results (d) sell access to the results to third parties so they can evaluate progress of models and the quality of their reasoning and (e) publish the positive results after a few months for free.

This is not a violation of the terms of service – I am just measuring the capabilities of the models and have them solve problems for me. Publishing the data isn’t a violation of the terms of service either.

Yet - by me publishing the positive results into the greater internet makes them...