Subject
29 entries
Bookmarks
PaLM: Scaling Language Modeling with Pathways
PaLM is Google's 540B parameter language model trained across 6144 TPU v4 chips using the Pathways distributed training system. It achieved breakthrough performance on multi-step reasoning and BIG-Bench, and documented discontinuous capability gains at scale — capabilities that emerged suddenly with more compute.
Leaked Google Document: "We Have No Moat, And Neither Does OpenAI"
Simon Willison's commentary on the leaked Google 'We Have No Moat' document, providing context and links to the SemiAnalysis publication. Willison frames the memo as significant for its candor about open-source AI's rapid quality trajectory.
Google "We Have No Moat, And Neither Does OpenAI"
A leaked internal Google document arguing that open-source AI will outcompete both Google and OpenAI — the thesis being that open-source models iterate faster, require no API fees, and are already approaching proprietary quality. One of the most influential strategic memos of the 2023 AI boom.
Google Research Deep Learning Tuning Playbook
Google Research's Deep Learning Tuning Playbook — a systematic guide to maximizing model performance through hyperparameter optimization. Written by Braxton Osting and team, it covers the science and art of tuning learning rates, batch sizes, regularization, and the full training configuration.
Solving Quantitative Reasoning Problems with Language Models (Minerva)
Lewkowycz et al. at Google Research introduce Minerva, a language model pretrained on general text and further trained on technical content that achieves state-of-the-art on quantitative reasoning benchmarks without external tools. It correctly answers nearly a third of undergraduate-level science problems — an early proof that domain-specific pretraining unlocks STEM reasoning at scale.
Google Talk to Books
Google's Talk to Books lets you search a large corpus of books using natural language statements, returning passages that respond semantically to your query. An early public demonstration of semantic search over a curated corpus, predating the vector database era.
Pathways: Asynchronous Distributed Dataflow for ML
Google's Pathways is a single-controller orchestration layer for ML accelerators that runs sharded asynchronous dataflow across thousands of TPUs while matching SPMD performance. Designed to break the MPI-style lockstep model so heterogeneous workloads like MoE, pipelining, and foundation-model multiplexing become first-class.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Switch Transformers (Fedus, Zoph, Shazeer 2021) scales language models to 1.6 trillion parameters using a simplified sparse Mixture of Experts architecture that routes each token to exactly one expert. It's the paper that made sparse MoE practical at scale and laid the architecture foundation for models like Mixtral and GPT-4.
MLOps Maturity Models: Google and Microsoft Frameworks
ZenML's overview of MLOps maturity models from Google and Microsoft — frameworks for thinking about how ML organizations can systematically improve how they develop and deploy models. Useful if you're trying to level up a team's ML practices from ad-hoc to automated.
PipelineDP: Differentially Private Data Aggregation
PipelineDP is an open-source framework from Google and OpenMined for differentially private data aggregation at scale — extract insights from large datasets while provably protecting individual privacy. Brings differential privacy out of academia and into data pipeline tooling.
Hidden Technical Debt in Machine Learning Systems
Sculley, Holt, Golovin et al. at Google extend the software engineering concept of technical debt to ML systems, identifying ML-specific forms that accumulate invisibly at the system level: boundary erosion, entanglement, hidden feedback loops, undeclared consumers, and data dependencies. The canonical paper explaining why ML systems are uniquely expensive to maintain.
Practical Advice for Analysis of Large, Complex Data Sets
Patrick Riley's practical guide to analyzing large, complex datasets from his years leading data science on Google Search logs. Covers sanity checks, stratification, and pitfalls that statistical theory alone won't protect you from.
Rules of Machine Learning: Best Practices for ML Engineering
Martin Zinkevich's 43-rule guide from Google on practical ML engineering, organized around the principle that most gains come from good features and solid infrastructure rather than clever algorithms. A pragmatic counterweight to academic ML papers — the kind of advice that separates production systems from demos.
Google Open-Sources Its Artificial Intelligence Engine
Wired's coverage of Google open-sourcing TensorFlow on November 9, 2015 — the moment that made production-grade deep learning infrastructure freely available to the world. A historical inflection point in the commoditization of AI tooling.
What Does It Take to Make Google Work at Scale?
A slide deck on what it takes to make Google's infrastructure work at scale — covering the distributed systems challenges, data storage, and engineering decisions behind running at internet scale. A useful systems design reference from before Designing Data-Intensive Applications became the canonical text.
Why We Partnered with Google to Expand to London
Galvanize's announcement of its London expansion in partnership with Google — its first international move, targeting the European tech education market. The Google partnership gave Galvanize credibility and potentially access to Google's London tech ecosystem.
The Mission to Bring Google's AI to the Rest of the World
Wired profiles Skymind, the startup founded by Adam Gibson to bring deep learning to enterprise Java shops — featuring Zipfian Academy's role in democratizing the field. A snapshot of 2014's optimism about deep learning accessibility before TensorFlow existed.
Google Turns to Machine Learning to Build a Better Data Centre
A 2014 report on Google applying machine learning to optimize data center cooling — one of the first public disclosures that Google was using neural networks to automate infrastructure decisions. DeepMind later published the full methodology in 2016.
Hadoop Creator: Google Is Living a Few Years in the Future
Doug Cutting (Hadoop creator) on Google living years ahead in infrastructure — the observation that Google's internal systems consistently anticipate what the rest of the industry will need, and then the open-source community builds it later.
The Datacenter as a Computer: Warehouse-Scale Machines
High Scalability's coverage of Google's 'The Datacenter as a Computer' second edition — Barroso and Hölzle's canonical text on warehouse-scale machine design. Defined the vocabulary and engineering tradeoffs for operating entire buildings as programmable compute platforms.
Google's Self-Driving Car Generates ~1GB Per Second
Bill Gross's 2013 tweet with an infographic showing Google's self-driving car generating ~1GB of sensor data per second — a striking illustration of the gap between human-scale perception and machine-scale data requirements for autonomous driving.
Rebecca Solnit: Google Invades San Francisco
Rebecca Solnit's LRB diary on Google's presence in San Francisco — arguing that tech wealth and private commuter buses were displacing long-term residents and erasing the city's identity as a refuge for artists and dissidents. The essay that crystallized the tech-gentrification debate.
Google Spanner: Globally Distributed Transactions (OSDI 2012)
Google's Spanner paper from OSDI 2012 — the design of Google's globally distributed SQL database with externally consistent transactions. TrueTime, the GPS/atomic-clock-based approach to distributed timestamps, is the paper's most memorable technical contribution.
Get That Job at Google
Steve Yegge's definitive guide on how to prepare for and pass Google software engineering interviews, written in 2008 and widely read through the 2010s. The advice is brutally practical: most candidates fail because they stopped practicing algorithms and data structures after college.
Sparrow Acquired by Google
Sparrow, the best Mac and iOS email client of its era, was acquihired by Google in July 2012 — immediately abandoning its user base without an acquirer to maintain the product. A defining moment in the 'acquihire kills good products' pattern.
Large-Scale Graph Computing at Google (Pregel)
Google Research's 2009 blog post introducing Pregel — their internal system for large-scale graph computation using a bulk-synchronous-parallel model. The post that launched the graph processing systems category and eventually spawned Apache Giraph, GraphX, and the whole vertex-centric computing tradition.
Google Opens BigQuery Data Analytics to All
GigaOM coverage of Google opening BigQuery to all developers in 2012 — the productization of Dremel as a cloud service. The moment when interactive SQL over petabyte datasets became a commercial product rather than a Google-internal tool.
Dremel: Interactive Analysis of Web-Scale Datasets
Google's Dremel paper — the system that enabled sub-second SQL queries over petabyte datasets via columnar storage and a multi-level serving tree. The direct precursor to BigQuery, and the inspiration behind Apache Parquet's nested record encoding.
Google Outs Google Drive: The Details
Google accidentally launched Google Drive via its French blog in April 2012 — a 5GB free cloud storage service directly competing with Dropbox. The comment in the original bookmark ('the sound of Dropbox share price hitting the floor') captures the competitive dread the moment triggered.