Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization. Play episode Play Related Sponsors Transcript Olimpiu Pop: Hello everybody, I'm Olimpiu Pop an InfoQ editor, and I have in front of me today, Edoardo Vacchi, one of the two Italians that usually talk about boxes inside of boxes. The other one is Andrea Peruffo.
Eduardo worked for a long period of time in the WebAssembly ecosystem, he did some spotting around other types of VMs, and today he's back at Red Hat and he's trying to make things easier in terms of Kubernetes and inference. It's a few people working on this, and I'm just one of them, but fair description. We're working on distributed inference on Kubernetes specifically, and that's called llm-d, and it's part of the VLLM family of projects. So, yes, it's super exciting to be back at Red Hat and work on the bleeding edge.
