Inference

Rakis Stability Test 1: Results from a Month of Running an AI Network

Rakis ran Stability Test 1: a month-long production test of a fully in-browser peer-to-peer AI inference network. 10 million tokens across ~2,000 nodes with embedding-based consensus for verification — early evidence that browser-native decentralized AI inference is viable.

AirLLM: 70B LLMs on a 4GB GPU

AirLLM runs 70B LLMs on a single 4GB GPU and 405B Llama3.1 on 8GB VRAM — without quantization, distillation, or pruning. It optimizes inference memory usage itself rather than compressing the model, preserving full model quality on consumer hardware.

Prompt Lookup Decoding: Speculative Decoding without a Draft Model

Prompt lookup decoding replaces the draft model in speculative decoding with simple n-gram string matching against the prompt itself — achieving 2.4x speedup on summarization and QA with zero quality loss. Works whenever output heavily references the input, which is most practical LLM tasks.

Outlines: Guided Text Generation

Outlines is a Python library for guided LLM text generation — constraining model outputs to match JSON schemas, regex patterns, or context-free grammars. It was one of the first production-quality structured generation libraries before OpenAI's own structured outputs feature.

MLC-LLM: LLMs on Every Device

MLC-LLM enables running large language models natively on any device — laptops, phones, browsers — without a server, by compiling models through the Apache TVM compiler stack. One of the earliest frameworks to make on-device LLM inference practical across diverse hardware.

Running Large Language Models in Production: Cohere's TIF

Cohere's post on their Transformer Inference Framework (TIF) — covering the systems challenges of serving large language models in production at scale. An early look at how LLM serving differs from traditional ML model serving and what optimizations matter.

Large Transformer Model Inference Optimization

Lilian Weng's comprehensive survey of transformer inference optimization techniques — covering quantization, distillation, pruning, efficient attention, speculative decoding, and hardware-level optimizations. The definitive reference for the topic as it stood in early 2023.

Wide Attention Is The Way Forward for Transformers

ONNX Stable Diffusion Exporter for Hugging Face Diffusers

The pull request adding ONNX export and an ONNX inference pipeline to Hugging Face Diffusers — enabling Stable Diffusion to run via ONNX Runtime on hardware accelerators beyond CUDA, including DirectML for Windows GPUs and optimized CPU inference.

Memorizing Transformers

ICLR 2022 spotlight proposing memory-augmented transformers that use approximate k-NN lookup into stored (key, value) pairs at inference time, without weight updates. Scales to 262K token memory with consistent perplexity improvements — an early approach to giving language models dynamic, test-time-updateable knowledge stores.

Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Networks

This paper demonstrates running deep neural network inference entirely on encrypted data using fully homomorphic encryption, so the server never sees plaintext inputs or outputs. It makes encrypted ML inference practical by combining FHE with approximation-friendly neural network architectures.

How fast can we perform a forward pass?

A deep technical investigation into the theoretical and practical speed limits of a single transformer forward pass, accounting for compute, memory bandwidth, and hardware constraints. The answer matters for inference costs, latency SLAs, and understanding where efficiency gains are actually possible.

TextSynth Playground

TextSynth is a web playground and API for running large language models efficiently — built by Fabrice Bellard (of ffmpeg and QuickJS fame) using his GGML-based inference engine. In 2022 it was one of the few accessible ways to experiment with GPT-J, GPT-NeoX, and similar open models.

Transformer Inference Arithmetic

Carol Chen (kipply) works through the arithmetic of transformer inference — compute vs. memory bandwidth, KV cache sizing, and how batch size shapes throughput/latency tradeoffs. Essential reference for anyone reasoning about LLM serving costs.