אודות פרק זה
Explore what it takes to run massive language models efficiently with Red Hat's Senior Principal Software Engineer in AI Engineering, Nick Hill. In this episode, we go behind the headlines to uncover the systems-level engineering making AI practical, focusing on the pivotal challenge of inference optimization and the transformative power of the vLLM open-source project.
Nick Hill shares his experiences working in AI including:
• The evolution of AI optimization, from early handcrafted systems like IBM Watson to the complex demands of today's generative AI.
• The critical role of open-source projects like vLLM in creating a common, efficient inference stack for diverse hardware platforms.
• Key innovations like PagedAttention that solve GPU memory fragmentation and manage the KV cache for scalable, high-throughput performance.
• How the open-source community is rapidly translating academic research into real-world, production-ready solutions for AI.
Join us to explore the infrastructure and optimization strategies making large-scale AI a reality. This conversation is essential for any technologist, engineer, or leader who wants to understand the how and why of AI performance. You’ll come away with a new appreciation for the clever, systems-level work required to build a truly scalable and open AI future.
Nick Hill shares his experiences working in AI including:
• The evolution of AI optimization, from early handcrafted systems like IBM Watson to the complex demands of today's generative AI.
• The critical role of open-source projects like vLLM in creating a common, efficient inference stack for diverse hardware platforms.
• Key innovations like PagedAttention that solve GPU memory fragmentation and manage the KV cache for scalable, high-throughput performance.
• How the open-source community is rapidly translating academic research into real-world, production-ready solutions for AI.
Join us to explore the infrastructure and optimization strategies making large-scale AI a reality. This conversation is essential for any technologist, engineer, or leader who wants to understand the how and why of AI performance. You’ll come away with a new appreciation for the clever, systems-level work required to build a truly scalable and open AI future.
אנגלית
ארצות הברית
בפרק זה
תעתיק 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
חיפוש פרקים ישנים
חפש פרקים קודמים של Technically Speaking with Chris Wright.
פרקים אחרים בפודקאסט הזה
As modern IT systems grow too complex for humans to effectively manage, there's growing interest in turning to autonomous AI agents for operations. While powerful, these agents introduce new challenges around trust, reliability, and control. To explore how to solve for this, Red Hat CTO Chris Wrig…
We often hear about AI's potential to generate text and code, but its application in healthcare and life sciences promises a more profound impact: solving our most fundamental challenges in disease and medicine. This isn't just about applying existing LLMs; it's about building entirely new foundat…
The software supply chain has always been a critical battleground, but AI introduces an exponential increase in scale and complexity. We are no longer just securing lines of code; we are now responsible for securing the models that generate it. This a new reality that shifts the entire attack surf…
Scaling LLM inference for production isn't just about adding more machines, it demands new intelligence in the infrastructure itself. In this episode, we're joined by Carlos Costa, Distinguished Engineer at IBM Research, a leader in large-scale compute and a key figure in the llm-d project. We dis…
הצהרה: הפודקאסט והגרפיקה המוטבעים בדף זה הם מ-Red Hat, שהוא רכושו של בעליו ואינו קשור או מאושר על ידי Listen Notes, Inc.
