Can a Windfall Trust ensure that no one gets left behind if AGI takes off?

AGI economy series - Nr. 5

Imagine a highly automated global economy in which billions of humans are jobless. Instead they live off charitable donations from gigantic AGI companies. Welcome to the Windfall Trust.

The Windfall Clause is an idea first proposed in 2020 by researchers from the Future of Humanity Institute at Oxford University to save livelihoods in a fast AGI take-off scenario by widely sharing the profits of AGI automation. The basic idea of the Windfall Clause is that leading AGI companies voluntarily sign a contract with an independent, non-profit Windfall Trust. This contract pre-commits the AGI companies to share a portion of their future profits with the trust, in case they ever reach astronomic levels. The Windfall Trust then has the mission to distribute these profits “for the good of humanity.”

While no AGI company has signed a Windfall Clause so far, considerations have moved forward with intentions of setting up an actual Windfall Trust. In July 2024 I attended a workshop on this topic organized by the Future of Life Institute and the Simon Institute for Longterm Governance. This post is in part a reflection on that, which highlights some of the challenges of the Windfall Clause and makes recommendations on how to set up a Windfall Trust.

Moravec’s Ark. Illustrated with ChatGPT.

1. The Windfall Clause as global social security

The core idea behind the Windfall Clause is to smooth the global, societal transition to advanced AI. Specifically, the authors argue that in an AGI take-off scenario as described above:

• a few AGI firms are likely to capture much of the future wealth from advanced AI

• there would be technological unemployment at a massive scale

• poorer countries would no longer develop via industrialization and the provision of cheap human labor, entrenching their economic disadvantage.

So, there is a need for global social security spending. However, this needs to be financed and the authors argue that:

• national taxation efforts overwhelmingly benefit the taxing nation’s citizens rather than being globally distributed.

• there is no taxation authority at the global / UN level that could then redistribute at a global level.

From that, and the assumption that most profits will go the US companies, the authors deduce that:

• the United States can presumably afford to take care of its citizens with corporate income taxes from its big tech companies—even if labor income taxes and social security contributions fall away

• In contrast, many other nations, and in particular developing nations, may not be able to provide adequate levels of social security.

The Windfall Trust is meant to help fill this gap with privately-funded social solidarity at the global scale. The following thresholds and sharing levels have been initially suggested by the Windfall Clause authors:

Source: Cullen O’Keefe et al. (2020). The Windfall Clause: Distributing the Benefits of AI for the Common Good. table 2.

GWP stands for Gross World Product or global GDP. This currently stands at about 105 trillion USD. So if a company that has signed the Windfall clause makes an annual profit of less than 105 billion USD (0.1% of GWP), it is not affected at all by it. As of 2023, only the state-owned oil producer Saudi Aramco, with a profit of 129 billion USD, crosses that threshold. As an example, if a signatory company would make a profit of 1 trillion USD (0.95% of GWP), it would give about 9 billion USD (1% of 895 billion USD) to the Windfall Trust. 

As the authors explain, a 1% marginal tax roughly corresponds to existing levels of corporate philanthropy and a 20% marginal tax roughly corresponds to a second layer of corporate income tax.

2. Why the Windfall Clause cannot replace public social security

a) The Windfall Clause is not triggered substantially in most scenarios

How many companies are likely to sign up for this clause? The authors suggest companies could do this for “general goodwill”. More cynically, it might signal to potential investors that a company thinks it might become very profitable in the future. Still, this may not be enough.

Even if OpenAI, Anthropic, and Deepmind would all sign the Windfall Clause, it’s not guaranteed that this is where the bulk of future AGI profits will be. So far, the foundation model layer seems competitive and has not been very profitable - almost all AI profits have been realised on the AI chip layer and, in particular, by NVIDIA’s near-monopoly. At least for the foreseeable future this is unlikely to change with AGI companies likely to reinvest potential profits into “blitzscaling”.

The suggested profit levels that trigger the Windfall Clause seem extremely high, requiring companies to both represent a significant share of world GDP and to have a high profit margin. For example, NVIDIA has a high profit margin but it’s still at a size where it would currently pay a 0% windfall tax. 

b) Commitments of firms under the Windfall Clause lack credibility

It is uncertain whether Windfall Trust obligations would really hold up in court. For example, it might be challenged by shareholders, such as the big cloud providers on which AGI companies depend. The obligations also lack a clear external enforcement mechanism and it’s not clear if a signatory company could not just, at any point, decide to unilaterally renege on this promise.

Even if a company sticks to its formal commitments, it seems fairly easy for a firm to shift or reduce book profits to stay below “Windfall Trust thresholds”.¹ For comparison: Taxes are a legal obligation for social solidarity backed by the monopoly of violence by the state. In comparison to taxes, the Windfall Clause is a pinky promise. Yet, big tech firms have managed to avoid paying taxes thanks to convoluted structures such as the famous “Double Irish with a Dutch Sandwich”, which routes profits through subsidiaries in Ireland and the Netherlands to exploit differences in tax laws. Overall, as

Dominik Hermle

points out, the track record of self-governance by AI companies is mixed at best.

c) The Windfall Trust cannot match social spending by states

Most developed countries spend between 15 and 30% of their GDP on various forms of social security. If activated, a Windfall Trust might reach something like 1% of GWP. The Windfall clause would not replace existing tax obligations, but be an addition to them. However, the Windfall Trust has such high activation thresholds that it may be primarily activated in post-Westphalian, technopolar scenarios in which antitrust enforcement has failed and AGI companies are more powerful than most states. In such a scenario many countries will lose revenue from labor income taxes and struggle to force AGI companies to pay taxes. If we only had the Windfall Trust as a social safety net that would reduce the share of world GDP spent on human welfare from about 20% to closer to 1%.² The only way to make such a net reduction in social solidarity seem generous in absolute terms as a global UBI is by assuming global GDP will grow more than 50-fold.³ 

The authors of the Windfall Clause argue “signatories would be more receptive to donations rather than taxation in order to reap the benefits of consumers, employees, and governments perceiving signatories to be socially responsible.” Another way to view this is that it shifts social responsibility from a stable, obligatory commitment (taxes) to a less predictable, voluntary act that comes with expectations of gratitude and some influence on how the available means are spent (philanthropic donations).

3. How to design a lasting Windfall Trust

The Windfall Trust can potentially play a role to fill some gaps in public social security systems, particularly in libertarian, technopolar AGI scenarios, where many public social security systems have been weakened. To ensure that a Windfall Trust works as intended I recommend three design choices:

a) Equity over promises

The voluntary commitments of AGI companies should include the immediate transfer of equity or exchange-traded shares to the Windfall Trust. That is a credible and irreversible commitment. Employees at an AI start-up don’t get promises of a shared future windfall, they get equity. If you want to have a fiduciary duty to humanity, give humanity shares. This creates an immediate cost for signing the Windfall clause, but that’s exactly the point. If AGI firms are not willing to give a few percent of their company to the trust now, it seems very unlikely that they would be willing to follow through on a sharing commitment, if it ever becomes worth a few trillion. Such a commitment can still be structured progressively to increase shares transferred upon reaching certain thresholds - but it should not start at zero. Generally, I would endorse Saffron Huang and Sam Manning’s argument for pre-distribution over redistribution.

b) Permanent fund

The Windfall Clause in its current form might be substantially activated if an AGI firm can make monopoly profits. However, monopolies may not last forever. The Trust may have just a few years of enormous windfall before prices for AGI services approach marginal cost. At the same time, there may be indefinite long-term, structural technological unemployment. To provide a sustainable insurance against this for centuries, the trust should follow Hartwick’s Rule and distribute dividends of a permanent fund rather than following a “pay as you go” model. The trust could hold concentrated bets on AGI companies in the beginning and then, if the portfolio has substantially increased in value,  gradually switch to a well-diversified wealth maintaining portfolio. The goal could then be to maintain an inflation-proof principal and to start paying out dividends in the form of a minimum income to the poorest of humanity.

c) Neutrality

If the Windfall Trust is meant to be a genuinely global organisation beholden to the welfare of the global population (and open to signatures from Chinese tech companies?) then it needs a credible neutrality that insulates it from geopolitical fights of the day. I would specifically encourage looking for lessons in the governance of the global domain name system through ICANN. Some powerful governments will predictably not like the governments of some of the poorest states in the world at some moments and might try to weaponize Windfall Trust payments for political purposes. To ensure the global credibility and the long-term stability that is needed to protect human welfare for many centuries in a future without work, neutrality should be a core principle of the organisation. The best way to ensure this is a host-state agreement with a stable, permanently neutral state that recognizes the trust as an international NGO and exempts it from sanctions.

A Windfall Trust alone is not enough

Corporate philanthropy should be viewed as a supplement to and not a replacement of public social security. The Windfall Trust could play a role as an insurance against some AGI scenarios. However, governments will not and should not accept these scenarios as the best possible outcome⁴ of a transition to an AGI economy. Rather, the Windfall Trust should be viewed as one tool amongst a broader set of policies. The overarching goal should be to help governments to adapt to an AGI economy, not to replace them. I will expand on ways to do this, such as international corporate tax governance, in future posts.

---

Thanks to Anna Yelizarova, Dominik Hermle, Robert Tracinski, Shreeda Segan and other Roots of Progress blog-building fellows for valuable feedback on a draft of this essay. All opinions and mistakes are mine.

1. Jan Leike (2020): “A core challenge when trying to design a windfall clause is that there is an incredibly strong incentive to find a loophole once the clause takes effect. If you run an organization who signed a windfall clause and in the future the unlikely comes to pass and you actually end up making $11 trillion in annual profits, it would be rational for you to spend up to $10 trillion on legal fees to try to get out of that clause just for that year; preferably in a way that doesn't cost you too much credibility. Companies are doing this already–this is why the big internet companies pay hardly any taxes.”

2. This is the extremized scenario for illustrative purposes when government social spending would go to zero due to inability to raise sufficient taxes from income tax and inability to raise taxes from large multinationals to make up for it. Governments could arguably still raise taxes from other sources such as VAT if they can maintain political stability. At the same time governments don’t just pay for social security but also for things such as public infrastructure and public services.

3. The founder and director of the Future of Humanity Institute Nick Bostrom explicitly makes an argument for a philantropy-based global UBI:

“In order for a $40,000 guaranteed basic annual income to be achieved with 5% of world GDP at 2018 population levels (of 7.6bn), world GDP would need to increase by a factor of 50 to 75, to 6 quadrillion (10^15) USD dollars. While 5% may sound like a high philanthropic rate, it is actually half of the average of the current rate of the ten richest Americans. While the required increase in economic productivity may seem large, it requires just six doublings of the world economy. Over the past century, doublings in world GDP per person have occurred roughly every 35 years. Advanced machine intelligence would likely lead to a substantial increase in the growth rate of wealth per (human) person. The economist Robin Hanson has argued that after the arrival of human-level machine intelligence, in the form of human brain emulations, doublings could be expected to occur every year or even month.”

An earlier draft stated: “In order for a $100,000 guaranteed basic annual income to be achieved with 1% of world GDP at current population levels, world GDP would need to increase to 71 quadrillion USD dollars. This is an increase of approximately 660 times the  current level when considering purchasing power parity, and 910 times the current level in nominal terms.”

4. The founder and director of the Future of Humanity Institute Nick Bostrom has suggested a philanthropy-financed global UBI as the best-case policy outcome for superintelligence.