Making Apache Kafka Diskless (with Filip Yonov & Josep Prat)
How do you retrofit a clustered data-processing system to use cheap commodity storage? That’s the big question in this episode as we look at one of the many attempts to build a version of Kafka that uses object storage services like S3 as its main disk, sacrificing a little latency for cheap, infinitely-scalable disks.There are several companies trying to walk down that road, and it’s clearly big business - one of them recently got bought out for a rumoured $250m. But one of them is actively trying to get those changes back into the community, as are pushing to make Apache Kafka speak object storage natively.Joining me to explain why and how are Josep Prat and Filip Yonov of Aiven. We break down what it takes to make Kafka’s storage layer optional on a per-topic basis, how they’re making sure it’s not a breaking change, and how they plan to get such a foundational feature merged.–Announcement Post: https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150Aiven’s (Temporary) Fork, Project Inkless: https://github.com/aiven/inkless/blob/main/docs/inkless/README.mdKafka Improvement Process (KIP) Articles: KIP-1150: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics KIP-1163: Diskless Core: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1163%3A+Diskless+Core KIP-1164: Topic Based Batch Coordinator: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1164%3A+Topic+Based+Batch+Coordinator KIP-1165: Object Compaction for Diskless: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1165%3A+Object+Compaction+for+DisklessSupport Developer Voices on Patreon: https://patreon.com/DeveloperVoicesSupport Developer Voices on YouTube: https://www.youtube.com/@developervoices/joinFilip on LinkedIn: https://www.linkedin.com/in/filipyonovJosep on LinkedIn: https://www.linkedin.com/in/jlprat/Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.socialKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
--------
1:29:29
Java's Cutting Edge Comeback (with Josh Long)
Java’s has been evolving faster than any 30 year old language has a right to do, and there’s probably no-one more pleased about it than my guest this week - Josh Long. He’s a Java & Kotlin programming, a JVM enthusiast in general, and an advocate for Spring, and he has chapters full of news about what’s been happening in Javaland over the past few years. Everything from new threading models to C interop changes, custom primitives to high performance computing and all the ways in which Java is modernising for age of AI workloads.If you’re out of touch with the latest in the JVM, or don’t know how much its changed, Josh’s brain is full of all the news you need to catch up.–Project Valhalla (Value Objects): https://openjdk.org/projects/valhalla/Project Panama (JVM’s new native code support): https://openjdk.org/projects/panama/Jextract: https://github.com/openjdk/jextractSpring Initializer: http://start.spring.io/Support Developer Voices on Patreon: https://patreon.com/DeveloperVoicesSupport Developer Voices on YouTube: https://www.youtube.com/@developervoices/joinKris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.socialKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
--------
1:24:29
The State & Future of Apache Kafka (with Anatoly Zelenin)
I’m joined this week by one of the authors of Apache Kafka In Action, to take a look at the state of Kafka, event systems & stream-processing technology. It’s an approach (and a whole market) that’s had at least a decade to mature, so how has it done? What does Kafka offer to developers and businesses, and which parts do they actually care about? What have streaming data systems promised and what have they actually delivered? What’s still left to build?–Apache Kafka in Action: https://www.manning.com/books/apache-kafka-in-actionPat Helland, Data on the Inside vs Data on the Outside: https://queue.acm.org/detail.cfm?id=3415014Out of the Tar Pit: https://curtclifton.net/papers/MoseleyMarks06a.pdfMartin Kleppmann, Turning the Database Inside-Out: https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.htmlData Mesh by Zhamak Dehghani: https://www.amazon.co.uk/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398Quix Streams: https://github.com/quixio/quix-streamsXTDB: https://xtdb.com/Support Developer Voices on Patreon: https://patreon.com/DeveloperVoicesSupport Developer Voices on YouTube: https://www.youtube.com/@developervoices/joinAnatoly’s Website: https://zelenin.de/Kris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins
--------
1:12:22
DataFusion - The Database Building Toolkit (with Andrew Lamb)
Building a database is a serious undertaking. There are just so many parts that you have to implement before you even get to a decent prototype, and so many hours of work before you could begin working on the ideas that would make your database unique. Apache DataFusion is a project that hopes to change all that, but building an extensible, composable toolkit of database pieces, which could let you build a viable database extremely quickly, and then innovate from that starting point. And even if you’re not building a database, it’s a fascinating project to explain how databases are built.Joining me to explain it all is Andrew Lamb, one of DataFusion’s core contributors, and he’s going to take us through the whole stack, how it’s built and how you could use it. Along the way we cover everything from who’s building interesting new databases and how you manage a large, open-source Rust project.–DataFusion Homepage: https://datafusion.apache.org/DataFusion on Github: https://github.com/apache/datafusionDataFusion Architecture (with diagrams!): https://youtu.be/NVKujPxwSBA?si=tw9ACxlbdpBuVsnv&t=1045Datalog: https://docs.racket-lang.org/datalog/Tokio: https://tokio.rs/Andrew’s Homepage: http://andrew.nerdnetworks.org/Andrew’s Blog Post about Tokio: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/Velox: https://velox-lib.io/Arroyo: https://www.arroyo.dev/Synnada: https://www.synnada.ai/LanceDB: https://lancedb.com/SDF+DBT: https://docs.sdf.com/integrations/dbt/integratingSupport Developer Voices on Patreon: https://patreon.com/DeveloperVoicesSupport Developer Voices on YouTube: https://www.youtube.com/@developervoices/joinKris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.socialKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Jupyter’s become an incredibly popular programming and data science tool, but how does it actually work? How have they built an interactive language execution engine? And if we understand the architecture, what else could it be used for?Joining me to look inside the Jupyter toolbox are Afshin Darian and Sylvain Corlay, two of Jupyters long-standing contributors and project-steerers. They’ve going to take us on a journey that starts with today’s userbase, goes through the execution protocol and ends with a look at what Jupyter will be in the future - an ambitious framework for interactive, collaborative applications and more.–Support Developer Voices on Patreon: https://patreon.com/DeveloperVoicesSupport Developer Voices on YouTube: https://www.youtube.com/@developervoices/joinJupyter Homepage: https://jupyter.org/Jupyter Xeus: https://github.com/jupyter-xeus/xeusJupyter AI: https://github.com/jupyterlab/jupyter-aiJupyter CAD: https://github.com/jupytercad/JupyterCADJupyter GIS: https://github.com/geojupyter/jupytergis/Jupyter GIS Announcement: https://blog.jupyter.org/real-time-collaboration-and-collaborative-editing-for-gis-workflows-with-jupyter-and-qgis-d25dbe2832a6QGIS: https://qgis.org/ZeroMQ: https://zeromq.org/Sylvain on LinkedIn: https://www.linkedin.com/in/sylvaincorlayDarian on LinkedIn: https://www.linkedin.com/in/afshindarianKris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.socialKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Deep-dive discussions with the smartest developers we know, explaining what they're working on, how they're trying to move the industry forward, and what we can learn from them.You might find the solution to your next architectural headache, pick up a new programming language, or just hear some good war stories from the frontline of technology.Join your host Kris Jenkins as we try to figure out what tomorrow's computing will look like the best way we know how - by listening directly to the developers' voices.
México