The Salience of Data
Subscribe
Let me begin with a thought experiment.
Imagine you know nothing about OpenAI’s or Anthropic’s financials. No ARR figures, and no valuation marks from the latest round. Now add one more condition, and it may be less hypothetical than it sounds: assume every frontier lab has access to a similar amount of compute, similar data, and similar talent.
Here is the question: sitting behind this veil of ignorance, how would you tell whether any of these model labs has a sustainable moat in the next 3-5 years? This question was actually much, much more difficult to answer couple of years ago. The fact that Anthropic was valued only $15 Billion in December 2023 may be indicative of how contrarian it was to bet on a model company and how challenging it was to articulate why these model companies would be able to outcompete the incumbent big tech companies when such big tech companies appeared to have almost unbounded access to capital, compute, talent, and data. To be fair, it is still quite challenging to articulate which particular model companies will end up dominating this layer. At least, today OpenAI and Anthropic have far more credible claims why they can have access to similar compute, and talent. It still seems silly to argue that OpenAI or Anthropic would have higher compute than Alphabet or Meta in the next 3-4 years. Talent in AI labs are also pretty mobile, so it’s hard to assume that as a source of any sustainable moat either. You can perhaps mention culture, but while I’m not denying culture as a source of potential moat, I always suspect investors mention about culture as a source of moat when it becomes difficult to pinpoint the source of sustainable moat.<br>How about data? You can legitimately argue one of the big reasons Anthropic has been such a resounding success is their focused bet on the best use case of these models: coding! And thanks to such a focused bet, they now have access to user data in coding which can beget to further improvement of the model. But even then, Anthropic’s coding model has found its greatest product market fit since December 2025. So, we aren’t even a year into this data flywheel moat and there may still be time for other model companies to respond pretty effectively to this moat. Codex is gaining ground and just yesterday, this tweet by Jukan made the case that xAI still has a pretty compelling shot at coding thanks to their acquisition of Cursor. Some excerpts from his tweet:<br>“One of the more interesting Grok bull cases I heard at ICML was this:
The core idea is that xAI may actually be better positioned than OpenAI Codex in the coding-agent market.
The reason Claude Code is currently leading in coding agents is not just model quality. Claude Code effectively pioneered the category at scale, which gave it one of the largest user pools in the industry. More users mean more real-world coding data. That data can then be used to improve Claude Code’s quality, which attracts even more users, creating a flywheel of more users, more data, and a better product.
Seen through this lens, xAI’s acquisition of Cursor starts to make a lot more sense.
Cursor likely has a much larger real-world user base and coding dataset than Codex. If xAI can effectively train on and leverage that data, the argument is that overtaking Codex may only be a matter of time.”<br>Given the success of coding use case, even Meta today entered the arena with their launch of Muse Spark 1.1 model.<br>Nonetheless, I do suspect data is likely the most reasonable explanation ex-ante why any particular model company may gain an upper hand over others while the rest of the inputs may be closer to commodity since all the relevant players will essentially have access to those. The fact that data is indeed the key source of long-term differentiation isn’t quite a new hypothesis. Back in June 2023, James Betker, a Research Engineer at OpenAI, in a piece titled “The “it” in AI models is the dataset” argued exactly this (emphasis mine):<br>“…I’ve trained a lot of generative models. More than anyone really has any right to train. As I’ve spent these hours observing the effects of tweaking various model configurations and hyperparameters, one thing that has struck me is the similarities in between all the training runs.
It’s becoming awfully clear to me that these models are truly approximating their datasets to an incredible degree.
…What this manifests as is – trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point.
…This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset.”<br>In a more recent piece, Will DePue, another OpenAI engineer who just left the company couple of months ago,...