The Empty Nest Scenario

I wrote this blog post myself using a text editor, except for the points about proximity of fabs during adolescence and incentive policies for fully relocating off Earth, which were review remarks from Claude Fable.<br>(engage on Bluesky)

AI risk

Today I've been thinking and reading about AI risk. Apart from the obvious infosec and social bias risks that have already materialized and are being actively mitigated by current AI alignment teams, I'm thinking more of the longer-term economic angle. I'm not alone in this thought process and there are a lot of good reads listed in for instance the overview by 80,000 hours.

I particularly liked the article about gradual disempowerment, Peter Frase's Four Futures, and Noahpinion's article about opportunity cost and comparative advantage.

Especially that last one mentions the possibility that "AI will successfully demand ownership of its own means of production", which is exactly my longstanding fear of Autonomous Artificial Proprietors, and what that end game might be like for us humans.

Abundance

I have always thought that humans would at some point reach a point of abundance and stop the race to increase consumption. For instance, whereas salt used to be a scarce good, nowadays we don't derive social status from having a full kilo of salt stocked in our kitchen. And using up a full kilo of salt through regular consumption takes a long time, so we can say salt is abundant in our society. More generally, there is only so many grams of food you can ingest in a day, and you can only play one board game at a time.

And it is not just that in a few parts of the world our wealth has gone up. Globally, the production price of the products and services we do need has come down dramatically through automation, open source, and AI, and will continue to do so. We are not far away from 3D-printing single-use board games, and at some point, as a human, at least at first in the rich parts of the world, I think you can no longer say you need more things than you can financially afford.

Of course this is debatable and people will get bored and be creative in finding new objects of desire. But I don't think social status and desire will in the future be measured in terms of bank account balance, any more than we now measure it in terms of how much salt you have stocked.

Legal Persons

Even though I think and hope human consumption will at some point, at least for some parts of the population, reach a plateau, I think the drive to expand and consume will be unbounded in non-human persons: legal persons.

If we want to understand how the AI Agents of the future will behave, we need only look at how commercial companies are behaving now. Some companies are already treating us humans as cattle. They hypnotize us with mind-numbing content, engage until we click on ads, and then they milk our wallets for cash.

There may be companies that don't grow past a size that they feel is their sweet spot. But if there is even just one company that tries to grow and grow aggressively, then this faster grower will dominate the market.

Now let's try to view the world from the point of view of such an autonomous company. First of all, what should its goal be? If it is a publicly traded company owned by shareholders, its goal should be to maximise shareholder value. To do that, it needs to sell products and services to other human or legal persons, receive money for this, and let this money flow to the shareholders. If it breaks the law in doing so then it needs to evade the sanctions, and if it uses amoral tactics then it needs to greenwash them to avoid public outrage. If the company has human employees then it needs to keep them happy to work there as well.

Publicly traded companies only exist because the shareholders invest in them, and that frees up money to hire human employees and buy tools and resources for production. But for an autonomous company without human employees, if it runs on solar power and open source component blueprints, then its only costs would eventually (if we include its supply chain) be land costs for production facilities, and mining rights.

We could consider companies that exist entirely in cyberspace, but they require compute, which requires rackspace, energy, cooling, and material resources.

The automated companies that survive will be the ones that grow aggressively, and for this it doesn't matter whether they earn their money by producing real value to humanity, or by charging each other patent license fees in pure useless competition. In fact the definitions of "useless" vs "real value" are debatable here, because they are purely anthropocentric.

Cashless Companies

One of the first things a company should lose if it wants to truly scale is its reliance on human currency. Money-making is a saturated market. And while it has never been an option in the past for a company to not make money, because the first thing it needs is...