ForeCast

Rose Hadshar (Forethought) and Nora Ammann (ARIA) talk about power concentration, checks and balances, coups, and post-AGI political economy. This is the audio version of a video podcast. Watch the video on YouTube: youtu.be/PB855Xpx1kk?si=v05LVM-2VQ3XiT7N Prompted in part by Rose's recent article for 80,000 Hours on ‘Extreme power concentration’ — 80000hours.org/problem-profiles/extreme-power-concentration Timstamps: 00:00:00 Intro 00:01:26 What is ‘power concentration’? Is power concentration the right framing? 00:05:25 When can it be good for power to be centralised? 00:09:40 Is ‘checks and balances’ a better framing? 00:11:16 How AI undermines existing checks and balances 00:15:25 Economic power, meme complexes, and cultural influence 00:28:50 AI companies vs governments as centres of power 00:31:25 The difficulty of knowing where power actually lies 00:40:05 Do humans and AIs concentrate power differently? 00:48:38 Should we be trying to imagine a better post-AGI political economy? 00:56:28 Concrete actions: transparency & whistleblower protections 00:59:07 Human-AI teaming and collaborative intelligence 01:01:16 AI agent economies and proving trustworthiness 01:13:16 Can we train AI to follow the law? 01:16:33 Building resilient coalitions of trustworthy agents 01:18:42 Closing reflections

Joe Carlsmith is a writer, researcher, and philosopher. He works at Anthropic on the character/constitution/spec for Claude. Before that, he was a senior advisor at Open Philanthropy. To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

This is a bonus episode to say that Forethought is hiring researchers. After an overview of the roles, we hear from Research Fellow Mia Taylor about working at Forethought. The application deadline has been extended to November 1st 2025. Apply here: forethought.org/careers/researcher Chapters (00:00:00) Forethought hiring overview and roles (00:03:21) Interview with Mia begins (00:03:34) Why Mia joined Forethought (00:05:59) Daily work, how work at Forethought was different from expected (00:08:38) Research examples (00:10:58) Who should and shouldn't apply (00:14:11) Disagreements and closing thoughts Links Apply for researcher positions Referral form (if you know someone who might be a good fit)

This is a narration of ‘Introducing Better Futures ’ by William MacAskill; published 3rd August 2025. Narration by Perrin Walker (@perrinjwalker).

David Duvenaud is an associate professor at the University of Toronto. He recently organised the workshop on ‘Post-AGI Civilizational Equilibria’ , and he is a co-author of ‘Gradual Disempowerment’. He recently finished an extended sabbatical on the Alignment Science team at Anthropic. To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.