Ultra-Fast LLM Inference: Streaming at 300+ tok/s with Groq LPUs

In generative AI applications, user perception of latency is dominated by two critical metrics: Time to First Token (TTFT) and Inter-Token Latency (ITL). When users submit complex prompts to traditional GPU-hosted models, a 2-to-4-second initial delay followed by a sluggish stream of 25–40 tokens per second creates friction and degrades conversational flow.

To deliver instantaneous user experiences—such as real-time code synthesis, low-latency agent tool calling, and conversational copilots—engineering teams are transitioning to Language Processing Units (LPUs) designed specifically for sequential autoregressive decoding.

This publication explores the underlying hardware architecture of LPUs versus GPUs, details an end-to-end Server-Sent Events (SSE) streaming pipeline in Node.js TypeScript, and outlines production resilience patterns.

1. Hardware Fundamentals: Why GPUs Stall on LLM Decoding

The fundamental bottleneck in autoregressive token generation is memory bandwidth, not matrix compute capacity:

+-------------------------------------------------------------------------+
|                              GPU Bottleneck                             |
|  Compute Cores (FLOPs) >> High Bandwidth Memory (HBM) Bus Capacity      |
|  Result: Matrix compute ALUs sit idle waiting for weights to be fetched |
+-------------------------------------------------------------------------+

+-------------------------------------------------------------------------+
|                              LPU Innovation                             |
|  Massive On-Chip Static RAM (SRAM) with 80+ TB/s internal bandwidth     |
|  Deterministic Instruction Scheduling: Zero cache misses, zero branch stalls |
+-------------------------------------------------------------------------+

1. The Prefill Phase (Compute-Bound): Processing the prompt tokens happens in parallel and leverages the GPU’s dense tensor cores efficiently.
2. The Decode Phase (Memory-Bound): Generating subsequent tokens is strictly sequential. Every single new token requires reloading dozens of gigabytes of model weights from High Bandwidth Memory (HBM) into SRAM cache.
3. Groq’s Tensor Streaming Processor (TSP): LPUs bypass external DRAM entirely during active layer operations. Models are distributed across interconnected LPU chips whose on-die SRAM provides over 80 Terabytes/sec of bidirectional bandwidth with zero cache-miss latency.

2. Benchmark Evaluation: GPU vs. LPU Performance

The table below contrasts standard cloud GPU inference instances against dedicated LPU inference racks running Llama 3.3 70B:

3. Production Node.js / TypeScript Streaming Pipeline

To deliver uninterrupted token streaming to client frontends without buffering delays or HTTP connection stalls, implement a Server-Sent Events (SSE) handler with explicit backpressure and chunk batching.

import { Router, type Request, type Response } from "express";
import Groq from "groq-sdk";

export const chatStreamRouter = Router();

const groq = new Groq({
  apiKey: process.env.GROQ_API_KEY,
  maxRetries: 3,
  timeout: 10000, // 10s strict timeout
});

interface StreamPayload {
  prompt: string;
  systemPrompt?: string;
  temperature?: number;
}

chatStreamRouter.post("/api/stream", async (req: Request, res: Response) => {
  const { prompt, systemPrompt, temperature = 0.2 }: StreamPayload = req.body;

  if (!prompt || typeof prompt !== "string") {
    return res.status(400).json({ error: "Missing or invalid prompt string." });
  }

  // 1. Establish SSE headers
  res.setHeader("Content-Type", "text/event-stream; charset=utf-8");
  res.setHeader("Cache-Control", "no-cache, no-transform");
  res.setHeader("Connection", "keep-alive");
  res.setHeader("X-Accel-Buffering", "no"); // Disable Nginx proxy buffering
  res.flushHeaders();

  // 2. Client disconnection handling
  let isClientConnected = true;
  req.on("close", () => {
    isClientConnected = false;
  });

  try {
    const stream = await groq.chat.completions.create({
      model: "openai/gpt-oss-120b",
      messages: [
        {
          role: "system",
          content: systemPrompt || "You are an expert distributed systems engineer providing precise solutions."
        },
        { role: "user", content: prompt }
      ],
      temperature,
      max_tokens: 2048,
      stream: true,
    });

    for await (const chunk of stream) {
      if (!isClientConnected) {
        break; // Stop inference consumption immediately if client aborts
      }

      const delta = chunk.choices[0]?.delta?.content || "";
      if (delta) {
        // Send structured event payload
        res.write(`data: ${JSON.stringify({ text: delta })}\n\n`);
      }
    }

    // Emit terminal completion packet
    if (isClientConnected) {
      res.write(`data: ${JSON.stringify({ done: true })}\n\n`);
      res.end();
    }
  } catch (error: any) {
    if (isClientConnected) {
      res.write(`data: ${JSON.stringify({ error: error.message || "Inference error" })}\n\n`);
      res.end();
    }
  }
});

4. Client-Side Chunk Reassembly with Error Recovery

Consuming a fast 300+ token/second stream requires non-blocking UI rendering. Updating React/DOM state on every single token causes layout thrashing. Instead, buffer chunks using requestAnimationFrame:

export async function consumeLpuStream(
  prompt: string, 
  onContentUpdate: (aggregatedText: string) => void
) {
  const response = await fetch("/api/stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`Stream request failed with status: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder("utf-8");
  let fullText = "";
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n\n");
    buffer = lines.pop() || ""; // Retain incomplete line

    for (const line of lines) {
      if (!line.startsWith("data: ")) continue;
      const jsonStr = line.slice(6).trim();
      
      try {
        const payload = JSON.parse(jsonStr);
        if (payload.done) return fullText;
        if (payload.text) {
          fullText += payload.text;
          onContentUpdate(fullText);
        }
      } catch (err) {
        console.warn("Skipping partial chunk JSON:", jsonStr);
      }
    }
  }

  return fullText;
}

5. Architectural Safeguards for Production Deployment

1. Proxy Buffering Elimination: Ensure your edge gateway (Cloudflare, Nginx, or AWS ALB) disables response buffering using X-Accel-Buffering: no. If buffering remains enabled, tokens will accumulate until the buffer fills, completely ruining the streaming UX.
2. Dynamic Degradation Fallbacks: Always configure a dual-provider fallback mechanism (e.g., Groq LPU primary $\rightarrow$ OpenAI/Anthropic secondary) with circuit breakers to safeguard against API quota exhaustion.
3. Local Cache Persistence: Cache completed dialogue turns in client IndexedDB (idb-keyval) to allow users to revisit responses instantly without re-generating tokens.