Subject
15 entries
GPU
Bookmarks
GPU Calculator for LLM Inference
An interactive calculator for estimating GPU requirements for LLM inference — input model size, quantization, batch size, and context length to get memory and throughput estimates. Practical first tool when planning self-hosted or cloud inference deployments.
CubeCL: Multi-Platform GPU Compute for Rust
CubeCL is a multi-platform GPU compute language extension for Rust — write one kernel that targets CUDA, ROCm, Vulkan/WebGPU, and Metal. The foundation for the Burn deep learning framework's backend abstraction.
Flash Attention: Derived and Coded from First Principles with Triton
A from-scratch implementation of Flash Attention using Triton (Python GPU programming), deriving the algorithm from first principles before coding it. Valuable for anyone wanting to understand how Flash Attention achieves its memory efficiency — learning by building rather than just using the library.
Building a Deep Learning Rig (Part 2)
Part 2 of building a home deep learning rig: upgrading to a Threadripper 1920X to support three RTX 3090s with full PCIe bandwidth. Total cost €2,379. Key lesson: GPU peer-to-peer DDP failures were fixed by downgrading the NVIDIA driver, not hardware changes.
AI's $200B Question
Sequoia Capital's 2023 analysis arguing that GPU capacity was being overbuilt relative to actual AI revenue, with $200B in GPU investment chasing far less in monetizable AI applications. A contrarian note during peak AI hype that proved partially prescient about the infrastructure overhang.
Compute Watch: LLM Compute Costs and GPU Availability Tracker
Compute Watch is a tracker for LLM compute costs and GPU availability — benchmarking inference costs across providers and tracking H100/A100 spot availability. Useful for anyone making infrastructure decisions around model serving costs.
Nvidia H100 GPUs: Supply and Demand
A detailed analysis of the Nvidia H100 GPU supply and demand situation in mid-2023 — how constrained supply was, where the demand was coming from, and what the bottlenecks were. Essential context for understanding the AI infrastructure market that year.
How Hugging Face Accelerate Runs Very Large Models
Hugging Face's technical guide to running very large models using Accelerate — covers device_map, model parallelism across GPUs, CPU offloading, and the mechanics of loading models that don't fit in a single GPU's VRAM. Essential reading for anyone self-hosting large LLMs.
Flash Attention: Training Examples
Flash Attention's training examples from Hazy Research — demonstrating how to use the memory-efficient attention implementation for training large transformers. Flash Attention became one of the most impactful algorithmic contributions to LLM training efficiency.
Banana: Serverless GPU Inference Hosting
Banana is a serverless GPU inference hosting platform for machine learning models — deploy your model in three steps, pay only for inference time. Targets the gap between spinning up a GPU VM yourself and the complexity of cloud ML serving infrastructure.
Piranha: A GPU Platform for Secure Computation
Piranha (USENIX Security 2022) is a GPU platform for secure multi-party computation that exploits GPU parallelism to accelerate MPC protocols by 10-15× over CPU implementations. It shows that GPU hardware can close the practical performance gap for privacy-preserving machine learning at scale.
A Hitchhiker's Guide to Distributed Training of Deep Neural Networks
Chahal, Grover, and Dey survey the algorithms and engineering techniques for distributed deep learning training, covering data parallelism, model parallelism, AllReduce strategies, gradient compression, and mixed precision. A practical reference for scaling training from single GPU to multi-node clusters.
Inferrd: Managed GPU Model Inference
Inferrd was a managed ML model serving platform with GPU support — deploy models via API without managing infrastructure. A 2021-era MLOps tool in the model deployment category that eventually got acquired.
MapGraph — GPU-Accelerated Graph Analytics
MapGraph is a high-performance graph analytics framework using GPU acceleration for large-scale graph algorithms. An early example of GPU-accelerated graph processing before NVIDIA's RAPIDS ecosystem made this mainstream.
Modern GPU
Sean Baxter's moderngpu library — a CUDA toolkit providing high-level primitives (sort, reduce, scan, join) for GPU programming. Published by NVIDIA Research, it demonstrated that expressive, composable GPU programming was achievable without sacrificing raw throughput.