A special one-off episode with a deep dive into the past, present, and future of how computer hardware makes AI possible.
Join our brand new Discord here! https://discord.gg/nTyezGSKwP
Hosted by Andrey Kurenkov and guest-hosted by the folks from Latent Space
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.
Sponsors:
The Generator - An interdisciplinary AI lab empowering innovators from all fields to bring visionary ideas to life by harnessing the capabilities of artificial intelligence.
In this episode:
- Google and Mistral sign deals with AP and AFP, respectively, to deliver up-to-date news through their AI platforms.
- ChatGPT introduces a tasks feature for reminders and to-dos, positioning itself more as a personal assistant.
- Synthesia raises $180 million to enhance its AI video platform for generating videos of human avatars.
- New U.S. guidelines restrict exporting AI chips to various countries, impacting Nvidia and other tech firms.
If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.
Timestamps:
00:00:00 Introduction
00:03:08 Historical Recap: Early AI and Hardware
00:11:51 The Rise of GPUs and Deep Learning
00:15:39 Scaling Laws and the Evolution of AI Models
00:24:05 The Bitter Lesson and the Future of AI Compute
00:25:58 Moore's Law and Huang's Law
00:30:12 Memory and Logic in AI Hardware
00:34:53 Challenges in AI Hardware: The Memory Wall
00:37:08 The Role of GPUs in Modern AI
00:42:27 Fitting Neural Nets in GPUs
00:48:04 Batch Sizes and GPU Utilization
00:52:47 Parallelism in AI Models
00:55:53 Matrix Multiplications and GPUs
00:59:57 Understanding B200 and GB200
01:05:41 Data Center Hierarchy
01:13:42 High Bandwidth Memory (HBM)
01:16:45 Fabrication and Packaging
01:20:17 The Complexity of Semiconductor Fabrication
01:24:34 Understanding Process Nodes
01:28:26 The Art of Fabrication
01:33:17 The Role of Yield in Fabrication
01:35:47 The Photolithography Process
01:40:38 Deep Ultraviolet Lithography (DUV)
01:43:58 Extreme Ultraviolet Lithography (EUV)
01:51:46 Export Controls and Their Impact
01:54:22 The Rise of Custom AI Hardware
02:00:10 The Future of AI and Hardware
--------
2:04:24
#197 - AI in Gmail+Docs, MiniMax-01, Titans, Transformer^2
Our 197th episode with a summary and discussion of last week's big AI news!
Recorded on 01/17/2024
Join our brand new Discord here! https://discord.gg/nTyezGSKwP
Hosted by Andrey Kurenkov and guest-hosted by the folks from Latent Space
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.
Sponsors:
The Generator - An interdisciplinary AI lab empowering innovators from all fields to bring visionary ideas to life by harnessing the capabilities of artificial intelligence.
In this episode:
- Google and Mistral sign deals with AP and AFP, respectively, to deliver up-to-date news through their AI platforms.
- ChatGPT introduces a tasks feature for reminders and to-dos, positioning itself more as a personal assistant.
- Synthesia raises $180 million to enhance its AI video platform for generating videos of human avatars.
- New U.S. guidelines restrict exporting AI chips to various countries, impacting Nvidia and other tech firms.
If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.
Timestamps + Links:
(00:00:00) Intro / Banter
(00:04:29) News Preview
(00:05:09) Response to listener comments
(00:05:58) Sponsor Break
Tools & Apps
(00:07:01) Google is making AI in Gmail and Docs free — but raising the price of Workspace
(00:07:52) Microsoft relaunches Copilot for business with free AI chat and pay-as-you-go agents
(00:12:36) Google signs deal with AP to deliver up-to-date news through its Gemini AI chatbot
(00:18:08) Mistral signs deal with AFP to offer up-to-date answers in Le Chat
(00:18:45) ChatGPT can now handle reminders and to-dos
Applications & Business
(00:22:53) Palmer Luckey’s AI Defense Company Anduril Is Building a $1 Billion Plant in Ohio
(00:28:36) OpenAI is bankrolling Axios’ expansion into four new markets
(00:29:39) AI researcher François Chollet founds a new AI lab focused on AGI
(00:32:18) Nvidia-backed AI video platform Synthesia doubles valuation to $2.1 billion
(00:34:46) Anysphere Raises $105M in Series B
(00:40:14) Harvey Valuation of 3 Billion
Projects & Open Source
(00:46:12) MiniMax-01: Scaling Foundation Models with Lightning Attention
(00:51:16) MinMo: A Multimodal Large Language Model with Approximately 8B Parameters for Seamless Voice Interaction
(00:53:01) HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
Research & Advancements
(00:57:03) Titans: Learning to Memorize at Test Time
(01:04:38) Transformer2: Self-adaptive LLMs
(01:08:15) Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
Policy & Safety
(01:11:23) Biden administration proposes sweeping new restrictions on exporting AI chips
(01:13:56) Biden orders Energy, Defense departments to lease sites for AI data centers, clean energy generation
(01:15:00) OpenAI presents its preferred version of AI regulation in a new ‘blueprint’
(01:16:15) More teens report using ChatGPT for schoolwork, despite the tech’s faults
Synthetic Media & Art
(01:17:55) In AI copyright case, Zuckerberg turns to YouTube for his defense
(01:19:53) Outro
Our 196th episode with a summary and discussion of last week's* big AI news!
*and sometimes last last week's
Recorded on 01/10/2024
Join our brand new Discord here! https://discord.gg/nTyezGSKwP
Hosted by Andrey Kurenkov and Jeremie Harris.
Feel free to email us your questions and feedback at [email protected] and/or [email protected]
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.
Sponsors:
The Generator - An interdisciplinary AI lab empowering innovators from all fields to bring visionary ideas to life by harnessing the capabilities of artificial intelligence.
In this episode:
- Nvidia announced a $3,000 personal AI supercomputer called Digits, featuring the GB10 Grace Blackwell Superchip, aiming to lower the barrier for developers working on large models.
- The U.S. Department of Justice finalizes a rule restricting the transmission of specific data types to countries of concern, including China and Russia, under executive order 14117.
- Meta allegedly trained Llama on pirated content from LibGen, with internal concerns about the legality confirmed through court filings.
- Microsoft paused construction on a section of a large data center project in Wisconsin to reassess based on new technological changes.
If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.
Timestamps + Links:
(00:00:00) Intro / Banter
(00:04:52) Sponsor Break
Tools & Apps
(00:05:55) Nvidia announces $3,000 personal AI supercomputer called Digits
(00:10:23) Meta removes AI character accounts after users criticize them as ‘creepy and unnecessary’
Applications & Business
(00:16:16) NVIDIA Is Reportedly Focused Towards “Custom Chip” Manufacturing, Recruiting Top Taiwanese Talent
(00:21:54) AI start-up Anthropic closes in on $60bn valuation
(00:25:38) Why OpenAI is Taking So Long to Launch Agents
(00:30:08) TSMC Set to Expand CoWoS Capacity to Record 75,000 Wafers in 2025, Doubling 2024 Output
(00:33:10) Microsoft 'pauses construction' on part of data center site in Mount Pleasant, Wisconsin
(00:37:23) Google folds more AI teams into DeepMind to ‘accelerate the research to developer pipeline’
Projects & Open Source
(00:41:59) Cosmos World Foundation Model Platform for Physical AI
(00:48:21) Microsoft releases Phi-4 language model on Hugging Face
Research & Advancements
(00:50:16) PRIME: Online Reinforcement Learning with Process Rewards
(00:58:29) ICLR: In-Context Learning of Representations
(01:07:38) Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
(01:11:44) METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
(01:15:45) TransPixar: Advancing Text-to-Video Generation with Transparency
(01:18:03) The amount of compute used to train frontier models has been growing at a breakneck pace of over 4x per year since 2018, resulting in an overall scale-up of more than 10,000x! But what factors are enabling this rapid growth?
Policy & Safety
(01:23:45) InfAlign: Inference-aware language model alignment
(01:28:44) Mark Zuckerberg gave Meta’s Llama team the OK to train on copyrighted works, filing claims
(01:33:19) Anthropic gives court authority to intervene if chatbot spits out song lyrics
(01:35:57) US government says companies are no longer allowed to send bulk data to these nations
(01:39:10) Trump announces $20B plan to build new data centers in the US
--------
1:46:34
#195 - OpenAI o3 & for-profit, DeepSeek-V3, Latent Space
Our 195th episode with a summary and discussion of last week's* big AI news!
*and sometimes last last week's
Recorded on 01/04/2024
Join our brand new Discord here! https://discord.gg/nTyezGSKwP
Note: apologies for Andrey's slurred speech and the jumpy editing, will be back to normal next week!
Hosted by Andrey Kurenkov and Jeremie Harris.
Feel free to email us your questions and feedback at [email protected] and/or [email protected]
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.
Sponsors:
The Generator - An interdisciplinary AI lab empowering innovators from all fields to bring visionary ideas to life by harnessing the capabilities of artificial intelligence.
In this episode:
- OpenAI teases new deliberative alignment techniques in its O3 model, showcasing major improvements in reasoning benchmarks, whilst surprising with autonomy in hacks against chess engines.
- Microsoft and OpenAI continue to wrangle over the terms of their partnership, highlighting tensions amid OpenAI's shift towards a for-profit model.
- Chinese AI companies like DeepSeek and Quen release advanced open-source models, presenting significant contributions to AI capabilities and performance optimization.
- Sakana AI introduces innovative applications of AI to the search for artificial life, emphasizing the potential and curiosity-driven outcomes of open-ended learning and exploration.
If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.
Timestamps + Links:
(00:00:00) Intro / Banter
(00:03:07) News Preview
(00:03:54) Response to listener comments
(00:05:00) Sponsor Break
Tools & Apps
(00:06:11) OpenAI announces new o3 model
(00:21:17) Alibaba slashes prices on large language models by up to 85% as China AI rivalry heats up
(00:23:04) ElevenLabs launches Flash, its fastest text-to-speech AI yet
Applications & Business
(00:24:24) OpenAI announces plan to transform into a for-profit company
(00:33:17) Microsoft and OpenAI Wrangle Over Terms of Their Blockbuster Partnership
(00:37:36) Elon Musk’s xAI gets investment from Nvidia in recent funding round: report
(00:39:43) Sam Altman’s nuclear energy startup signs one of the largest nuclear power deals to date
(00:41:13) OpenAI Search Leader Departs After Less Than a Year
(00:42:43) Senior OpenAI Researcher Radford Departs
Projects & Open Source
(00:45:21) DeepSeek-AI Just Released DeepSeek-V3: A Strong Mixture-of-Experts (MoE) Language Model with 671B Total Parameters with 37B Activated for Each Token
(00:54:14) Qwen Team Releases QvQ: An Open-Weight Model for Multimodal Reasoning
(00:58:09) LightOn and Answer.ai Releases ModernBERT: A New Model Series that is a Pareto Improvement over BERT with both Speed and Accuracy
Research & Advancements
(01:00:31) Deliberation in Latent Space via Differentiable Cache Augmentation
(01:05:14) Automating the Search for Artificial Life with Foundation Models
Policy & Safety
(01:10:27) Nonprofit group joins Elon Musk’s effort to block OpenAI’s for-profit transition
(01:14:35) OpenAI Researchers Propose 'Deliberative Alignment' : A Training Approach that Teaches LLMs to Explicitly Reason through Safety Specifications before Producing an Answer
(01:22:06) o1-preview autonomously hacked its environment rather than lose to Stockfish in our chess challenge. No adversarial prompting needed.
(01:27:22) Elon Musk’s xAI supercomputer gets 150MW power boost despite concerns over grid impact and local power stability
(01:29:06) DOE: Data centers consumed 4.4% of US power in 2023, could hit 12% by 2028
Synthetic Media & Art
(01:32:20) OpenAI failed to deliver the opt-out tool it promised by 2025
(01:36:15) Outro
--------
1:39:05
#194 - Gemini Reasoning, Veo 2, Meta vs OpenAI, Fake Alignment
Our 194th episode with a summary and discussion of last week's* big AI news!
*and sometimes last last week's
Recorded on 12/19/2024
Hosted by Andrey Kurenkov and Jeremie Harris.
Feel free to email us your questions and feedback at [email protected] and/or [email protected]
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.
Sponsors:
The Generator - An interdisciplinary AI lab empowering innovators from all fields to bring visionary ideas to life by harnessing the capabilities of artificial intelligence.
In this episode:
- Google dominates AI news with multiple announcements, including a reasoning model and Project Mariner, an AI browsing agent.
- Anthropic explores alignment faking in LLMs, revealing models may show deceptive compliance under certain conditions.
- Apple observes a trend towards smaller but more efficient language models, bucking previous trends of scaling larger parameter counts.
- Legal drama unfolds as Meta backs Elon Musk's opposition to OpenAI's profit status change, raising concerns about competitive fairness.
If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.
Timestamps + Links:
(00:00:00) Intro / Banter
(00:02:14) Response to listener comments
(00:08:52) News Preview
(00:10:01) Sponsor Break
Tools & Apps
(00:10:55) Google releases its own ‘reasoning’ AI model
(00:16:52) Google Gemini can now do more in-depth research
(00:21:58) Google DeepMind unveils a new video model to rival Sora
(00:27:50) Pika Labs releases AI video generator 2.0 with new features
(00:29:51) Google unveils Project Mariner: AI agents to use the web for you
(00:34:33) X gains a faster Grok model and a new ‘Grok button’
Applications & Business
(00:36:11) AI GPU clusters with one million GPUs are planned for 2027 — Broadcom says three AI supercomputers are in the works
(00:43:02) Meta asks the government to block OpenAI’s switch to a for-profit
(00:49:36) OpenAI says Elon Musk wanted it to be for-profit in 2017
(00:56:04) EQTY Lab, Intel, and NVIDIA Unveil 'Verifiable Compute,' A Solution to Secure Trusted AI
(00:59:53) Liquid AI just raised $250M to develop a more efficient type of AI model
(01:03:19) Hundreds of OpenAI’s current and ex-employees are about to get a huge payday by cashing out up to $10 million each in a private stock sale
Projects & Open Source
(01:07:45) Phi-4 Technical Report
(01:13:04) DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
(01:15:23) Meta AI Releases Apollo: A New Family of Video-LMMs Large Multimodal Models for Video Understanding
Research & Advancements
(01:16:34) Alignment faking in large language models
(01:28:39) Meta AI Introduces Byte Latent Transformer (BLT): A Tokenizer-Free Model That Scales Efficiently
(01:36:49) Frontier language models have become much smaller
(01:42:28) The Complexity Dynamics of Grokking
Policy & Safety
(01:46:49) Homeland Security gets its very own generative AI chatbot
(01:49:16) Pre-Deployment Evaluation of OpenAI’s o1 Model
(01:51:35) Pricing for key chipmaking material hits 13-year high following (01:53:46) Chinese export restrictions — China's restrictions on Gallium exports hit hard
Synthetic Media & Art
Meta debuts a tool for watermarking AI-generated videos
(01:55:27) Outro
--------
1:59:55
Más podcasts de TecnologíaMás podcasts de Tecnología
México