Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts

Lightbits Labs, the company that invented NVMe over TCP, is moving into inference software with Inferra, a KV cache orchestration engine that makes its public debut tomorrow, September 15, at the AI Infra Summit in Santa Clara. The software virtualizes GPU memory across DRAM and NVMe storage tiers and turns the KV cache into a persistent data layer, so attention states for long-context and multi-session workloads don’t have to fit in HBM or be recomputed when they fall out. Lightbits claims up to 16x more concurrent inference sessions on existing GPUs, a greater than 100x latency improvement from prefetching attention states in place of recomputing them, and context windows up to 10 million tokens, and says the engine has already run in customer beta programs, including one with OVH.

The bottleneck Inferra targets is the one we walked through in our KV cache offload to flash piece: as context lengths grow and sessions multiply, the KV cache outgrows GPU memory, and the serving stack either evicts it and recomputes the prefill later or caps how many sessions a GPU can hold. Both paths burn GPU cycles on work that has already been done. Inferra keeps the cache alive across memory and storage tiers, and a predictive prefetcher pulls attention states back toward the GPU ahead of when the model needs them. Lightbits says that approach drives down both Time-to-First-Token (TTFT) and Time Per Output Token (TPOT), and the 100x figure it quotes is the latency improvement against recomputing those states from scratch.

Lightbits’ block diagram lays out the four pieces: an intelligent prefetcher, a tiering manager, a log-structured KV store, and security and isolation engines, sitting between the serving framework above and a standard NVMe SSD pool below. The company lists vLLM, TensorRT, and SGLang as supported serving frameworks and says the engine is GPU-and SSD-agnostic. The log-structured store is what makes a flash tier practical for a cache that changes constantly, since it appends updates sequentially and keeps small random writes off the drives, and we looked at how a 3 DWPD drive holds up in that role in our Solidigm D7-PS1030 review. On the multi-tenant side, Lightbits describes secure tenant isolation and intelligent cache management across shared inference infrastructure, with encrypted transfer between tiers, which is what lets a neocloud hand one KV pool to many customers with consistent SLAs.

Avigdor Willenz, co-founder and chairman of Lightbits Labs, framed the launch as a break from storage built for training pipelines. Inference “is a completely different paradigm,” he said, and “retrofitting legacy training and storage systems to solve the KV cache bottleneck simply doesn’t work. It was built for a different era. We engineered Inferra from the ground up specifically to solve the GPU efficiency problem, and it has already proven successful in customer beta programs.”

OVH is the named beta customer. “With Inferra’s intelligent KV cache tiering, we were able to demonstrate substantial GPU utilization gains, paving the way to providing our customers a more scalable and cost-effective foundation for their AI agent and RAG workloads,” said Yaniv Fdida, chief product and technology officer at OVH. Solidigm ran the engine in its AI Central Lab on D7-PS1010 drives, and Avi Shetty, the company’s VP of ecosystem, solutions, and market enablement, said the results “showcased how network-attached storage with intelligent software layer virtualization can break through the memory wall for large context, agentic AI.” That pairing of a network-attached NVMe pool with a software tier is Lightbits’ home turf; its LightOS block storage has been built on NVMe/TCP since 2019.

Ramesh Chettuvetty, senior vice president of AI product and business at Lightbits, called Inferra “the world’s first KV cache acceleration engine that eliminates idle GPUs and power context windows of up to 10M tokens on commodity hardware,” and said it “delivers instant payback and generates net positive savings from day one.” Those are vendor claims ahead of any independent testing, and the release doesn’t give pricing, packaging, or a general availability date; the engine is in customer beta today. Lightbits is running live demos of the session-density and latency results at Booth 219 during the summit, and has posted a Pod Efficiency Analyzer for teams that want to model the GPU savings against their own cluster before booking one.

The post Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts appeared first on StorageReview.com.