Subject
16 entries
Optimization
Bookmarks
Dynamic Programming: QuantEcon textbook
QuantEcon's open-source Dynamic Programming textbook teaches the mathematical foundations of sequential decision-making with Python and Julia implementations. Covers Bellman equations, value function iteration, and stochastic control — the tools behind modern RL and macro models alike.
Flash Attention: Derived and Coded from First Principles with Triton
A from-scratch implementation of Flash Attention using Triton (Python GPU programming), deriving the algorithm from first principles before coding it. Valuable for anyone wanting to understand how Flash Attention achieves its memory efficiency — learning by building rather than just using the library.
TensorZero
TensorZero is a self-hosted LLMOps platform that unifies gateway, observability, evaluation, optimization, and A/B testing in one Rust-built system. The feedback loop: production inference data flows back into prompt and model improvement automatically.
Cohere Prompt Tuner: Automated Prompt Optimization
Cohere's Prompt Tuner automatically improves prompts for their models by testing variations and selecting what performs best — prompt optimization as a first-class product feature rather than a manual art.
Large Transformer Model Inference Optimization
Lilian Weng's comprehensive survey of transformer inference optimization techniques — covering quantization, distillation, pruning, efficient attention, speculative decoding, and hardware-level optimizations. The definitive reference for the topic as it stood in early 2023.
Learning from History for Byzantine Robust Optimization
Karimireddy, He, and Jaggi (EPFL, arXiv:2012.10333, 2021) propose using historical gradient information to detect and filter Byzantine workers in distributed training, achieving near-optimal convergence even with a constant fraction of corrupt workers. The historical approach is notable because it breaks the fundamental limitation of single-round Byzantine filters without requiring cryptographic overhead.
Asynchronous Decentralized SGD with Quantized and Local Updates
SwarmSGD is an asynchronous decentralized optimization algorithm that provably converges when combining gossip communication, gradient quantization, and local update steps simultaneously across heterogeneous data distributions. It achieves performance comparable to large-batch SGD on supercomputing systems while reducing communication overhead.
PioSOLVER: Game Theory Optimal Poker Solver
PioSOLVER is the industry-standard GTO (Game Theory Optimal) solver for poker — computes Nash equilibrium strategies for heads-up and multi-way spots using CFR algorithms. Used by professional players to study theoretically unexploitable play.
An Introduction to Gradient Descent and Linear Regression
A clear walkthrough of gradient descent applied to linear regression, with code — a good foundational tutorial connecting the mathematical update rule to a concrete implementation. One of the most-linked introductions to the algorithm for newcomers to ML.
Optimism in the Face of Uncertainty: the UCB1 Algorithm
Jeremy Kun's accessible treatment of the UCB1 algorithm — the principle of 'optimism in the face of uncertainty' formalized as a bandit algorithm with proven regret bounds. Shows why adding a confidence bonus to estimated rewards elegantly solves the exploration-exploitation tradeoff.
A Book About Bandit Algorithms
John Myles White's free book on bandit algorithms for website optimization — covers epsilon-greedy, softmax, UCB, and Thompson Sampling with practical web application examples. An accessible bridge from theory to product experimentation.
A/B Testing with Bayesian Bandits in Google Analytics
FastML walkthrough of Google Analytics Content Experiments — which use a multi-armed bandit algorithm instead of fixed 50/50 splits to dynamically allocate traffic toward better-performing variants. A practical introduction to bandit-based online optimization.
I'm a Bandit — Sébastien Bubeck's Optimization Blog
I'm a Bandit is Sébastien Bubeck's research blog on optimization and online learning — covering convex optimization, bandit algorithms, and randomized methods at a level between textbook and paper. One of the best technical blogs in theoretical ML/optimization.
Speeding Up Your Python Code
Max Burstein's practical guide to Python performance — profiling-driven optimization covering list comprehensions, generators, local variable access, string concatenation, and C extension use. A good 2013-era reference for the pragmatic Python performance techniques.
Machine Learning: Genetic Algorithms in JavaScript
Burak Kanber's accessible tutorial implementing genetic algorithms in JavaScript from scratch — part of a machine learning series aimed at web developers. One of the early 'ML for JS devs' resources that predated the current wave of browser-based ML by several years.
The Perfect Milk Machine: How Big Data Transformed the Dairy Industry
Alexis Madrigal's Atlantic piece on how the dairy industry used decades of genetic and performance data to engineer Holstein cows into radically more efficient milk producers. The best early example of big data optimization applied to a non-tech domain.