Dario Amodei's stance on open weights is self-serving and short-sighted | Jan Iłowski

Leaders are often conflicted people. Marcus Aurelius wrote about the common humanity of all rational beings in “Meditations” while commanding invasions. Abraham Lincoln thought of slavery as morally wrong, yet his “paramount object” of preserving the Union made him initially resist immediate abolition.<br>In an interview with The Economist last week, Elon Musk said he can go “from exhilaration to terror regarding AI” during a single day. Dario Amodei seems to be conflicted as well. In his recent memo on open-weight AI models, the best of which are currently trained in China, he wrote in bold lettering that he had never advocated for a ban on them, yet in the very next sentence began adding asterisks to that statement. The New York Times reported last week that Anthropic privately lobbied for increased restrictions on Chinese open-weight AI models.<br>Dario Amodei writes that he only considers open-weight AI models that “don’t have dangerous capabilities” a public good. Later in the memo, he supports mandatory, government-organized safety testing for all sufficiently capable models, arguing that open-weight models are intrinsically harder to make safe because their safeguards can be removed. Proprietary models are inherently safer in his telling because their use is monitored and access to them can be withdrawn at any time. This line of thinking suggests that open weights are acceptable provided they remain below some government-defined capability threshold. If they are too capable, their use should be monitored or restricted, and since their use can’t be monitored, they should be curtailed by law.<br>Anthropic sells controlled access to centrally hosted models. A regulatory system built around expensive pre-release evaluations, continuous usage monitoring, identity verification, and revocable access naturally favours large providers of proprietary models. Anthropic already possesses the compliance, legal, and evaluation infrastructure needed to satisfy such a regime. A university lab, startup, or community does not, even if they somehow manage to train a sufficiently capable model.<br>Moreover, nominally applying the same test to closed and open-weight models is not genuinely symmetrical. Anthropic can control Claude’s safeguards. An open model must effectively be evaluated under its worst plausible modification because users can work around its built-in restrictions. The open model therefore faces an inherently harder standard. This fits the definition of regulatory capture: regulating the industry around the incumbent’s operating model while describing the rules as neutral.<br>Dario Amodei pays little attention to some of the main advantages of open-weight models. Companies and institutions can reliably run their own AI without relying on a handful of vendors with volatile policies and less-than-predictable billing and access conditions. Researchers can inspect, study (or at least try to), and fine-tune them.<br>Protectionism, especially restrictions on imports, will always reduce competitiveness and harm the companies it aims to “protect” as well as the country in the long term. If the current U.S. administration heeds the advice of Anthropic’s chief, the result will be a reduction in the West’s global competitiveness in AI and another step in ceding our civilizational dominance in this emerging technology. China will continue releasing new open-weight models, causing developers and tooling to adapt to an open Chinese stack instead of the restrictive and less competitive American offering.<br>Furthermore, export controls, for which Dario Amodei openly advocates, are already prompting China to build its own chips. China is, of course, behind the West, but not as much as you may think. Instead of adopting a Western technology stack and relying on our technology to build its AI models, China will increasingly rely on its own chips, reducing Western leverage.<br>Another of Dario Amodei’s proposed “crackdowns” targets “industrial-scale distillation”. He concedes that “distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US”, but argues that what matters is that it is “backed by an authoritarian state seeking to overtake the US at the frontier”. I understand his frustration that Chinese labs may be capturing some of his profits by offering cheap distillations of Claude. However, I think he shouldn’t dress up that (serious) commercial concern as the grander claim that Chinese AI will overtake the US by means of distillation.<br>I think Dario Amodei is sincerely afraid of a possible future danger from AI. I agree that there should be an industry or government body that briefly evaluates AI models before release, perhaps based on what Demis Hassabis has recently suggested. I also think the ideals of trust and safety on which Anthropic was founded are not fully thought through or perhaps not truly valued enough,...