The AI Speed War: How French Startup Kog Is Rewriting the Rules of GPU Efficiency

The race for faster AI inference is heating up, and while markets gave Cerebras and its purpose-built chips a warm welcome during its IPO debut in May, French startup Kog is betting that the real revolution lies in squeezing more performance out of the conventional GPUs companies already own.

A Software Revolution Challenging Hardware Limits

Kog hit the front page of Hacker News with a tech preview proving that "extremely fast single-request decoding is possible" on standard datacenter GPUs like the AMD MI300X and Nvidia H200 used in their demo. The startup's promise is to unlock new capabilities on existing hardware through software optimization, effectively lowering costs.

Capitalizing on the Inference Bottleneck

With inference speed and costs now a critical bottleneck, CEO Gaël Delalleau revealed that the promise of optimizing existing hardware attracted more than just onlookers, garnering 200 tangible business leads. This approach targets customers put off by delays in professional AI workflows.

■Software engineering is expected to be the first major use case.

■Anthropic’s premium pricing for Claude’s Fast Mode highlights the market's thirst for speed.

■For design partners generating games and apps, faster outcomes directly translate to revenue.

A "Hacker" Approach Rooted in Physics and Cybersecurity

Kog’s deep-level focus stems from the unique background of its CEO, who studied solid-state physics at École Polytechnique and worked in offensive cybersecurity. The team adopts a mindset to understand the laws of physics and the GPU, reverse-engineering down to assembly language.

■The demo achieved an impressive 3,000 tokens per second using the open-source Laneformer 2B (2 billion parameters).

■The company aims to prove this approach works for larger LLMs to meet demand.

■The hands-on approach requires dedicating weeks or months to research for every new GPU.

European Tech Sovereignty and the Path to Scale

Operating with a team of 11 people, Kog faces limits on the number of chips it can support but plans to feed its methodology into agent-based pipelines. As Europe seeks to build its own capabilities, support from Scaleway, Bpifrance, and French Tech 2030 provides tailwinds.

Markets are reacting to this with a question about whether Moore's Law is limited by hardware speed. Kog's ability to "hack" GPU memory bandwidth creates an interesting paradox for giants like Nvidia and AMD. While the race for specialized chips (ASICs) continues, software optimization extends the life of current inventory. If Kog can demonstrate the promised 10x speed increase in large models by September as planned, it could significantly improve the CAPEX efficiency of AI infrastructure investments, positively impacting tech stock margins. However, such hardware-dependent optimization highlights that software is becoming a strategic weapon in the China-US chip war.