ForeCast

Rose Hadshar is a researcher at Forethought. She discusses:

Why governments might pursue an international AGI government

The chance that the first AGI developer becomes a de facto world government

The "Intelsat model" – treating AGI as commercial infrastructure rather than a weapon or science project

Membership and voting structures that might make a US-led coalition viable

The "AGI convention" proposal

Watch the video version on YouTube.

Read a full transcript here.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

Tom Davidson and Will MacAskill are both researchers at Forethought.

They discuss:

What is the industrial explosion?

Why the case for recursive self-improvement is stronger for physical industry than for software

How fast the physical economy could grow, the case for weekly doubling times, and limits from natural resources

Why authoritarian regimes might have a structural advantage in the industrial explosion

Could a leading country outgrow the entire world to achieve decisive dominance?

Why does the industrial explosion get ~1% of the attention of the intelligence explosion?

You can read a full transcript here.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

Lukas Finnveden is a Research Analyst at Redwood Research. We talk about making deals with early scheming AIs: paying them to cooperate rather than take over.

You can read a full transcript here.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

Rose Hadshar (Forethought) and Nora Ammann (ARIA) talk about power concentration, checks and balances, coups, and post-AGI political economy.

You can read a full transcript here.

This is the audio version of a video podcast. Watch the video on YouTube: youtu.be/PB855Xpx1kk?si=v05LVM-2VQ3XiT7N

Prompted in part by Rose's recent article for 80,000 Hours on ‘Extreme power concentration’ — 80000hours.org/problem-profiles/extreme-power-concentration

Timstamps:

00:00:00 Intro

00:01:26 What is ‘power concentration’? Is power concentration the right framing?

00:05:25 When can it be good for power to be centralised?

00:09:40 Is ‘checks and balances’ a better framing?

00:11:16 How AI undermines existing checks and balances

00:15:25 Economic power, meme complexes, and cultural influence

00:28:50 AI companies vs governments as centres of power

00:31:25 The difficulty of knowing where power actually lies

00:40:05 Do humans and AIs concentrate power differently?

00:48:38 Should we be trying to imagine a better post-AGI political economy?

00:56:28 Concrete actions: transparency & whistleblower protections

00:59:07 Human-AI teaming and collaborative intelligence

01:01:16 AI agent economies and proving trustworthiness

01:13:16 Can we train AI to follow the law?

01:16:33 Building resilient coalitions of trustworthy agents

01:18:42 Closing reflections

Joe Carlsmith is a writer, researcher, and philosopher. He works at Anthropic on the character/constitution/spec for Claude. Before that, he was a senior advisor at Open Philanthropy.

You can read a full transcript here.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.