NLP

Reframing Instructional Prompts to GPTk's Language

ACL 2022 Findings paper showing that manually reframing instructional prompts — decomposing tasks, itemizing steps, adding positive examples — yields 6–12% performance gains on GPT-2 and GPT-3. The key insight is that models respond better to concrete, step-by-step instructions than to long abstract descriptions.

LangExtract: Gemini-powered structured information extraction

LangExtract is Google's open-source Python library for structured information extraction from unstructured text using Gemini's controlled generation. Maps every extracted element back to exact character positions in source text.

Ethos: Hacker News analysis with vector embeddings

Ethos analyzes Hacker News discussions using vector embeddings to surface concepts, entities, sentiment, and discourse patterns. Open-source, from devrupt.io.

annotateai

annotateai is a Python package that uses LLMs to automatically annotate research papers — highlighting key claims, methodology sections, results, and limitations. Useful for quickly mapping what a paper is about before reading it in full.

Dense-X-Retrieval: Proposition-Level RAG

Dense-X-Retrieval is a LlamaIndex pack implementing proposition-level retrieval — splitting documents into atomic factual propositions rather than chunks, then retrieving at proposition granularity. Improves precision by matching query semantics at a finer level than paragraph chunks.

Optimal Chunk Size for Large Document Summarization

Vectify AI introduces a method to automatically determine the optimal chunk size for large document summarization with LLMs — rather than fixed-size chunking, the approach finds the chunk granularity that maximizes summary quality for a given document type.

INDRA: Automated Biomedical Knowledge Assembly

INDRA (Integrated Network and Dynamical Reasoning Assembler) is an automated knowledge assembly system that reads biomedical literature and databases and produces causal graphs and dynamical models. A long-running academic system for automated scientific knowledge extraction.

BERTopic: BERT-Based Topic Modeling

BERTopic is the leading open-source topic modeling library using sentence embeddings and clustering rather than word co-occurrence statistics — produces coherent, human-readable topics that LDA-style models often can't. The changelog tracks its evolution as the library added new backends and features.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Lewis et al. (Facebook AI, 2020) introduce Retrieval-Augmented Generation, a hybrid architecture that combines dense passage retrieval with seq2seq generation to ground language model outputs in a non-parametric knowledge store. RAG defined the template that most production knowledge-grounded LLM systems follow today.

Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation

NeurIPS 2022 paper investigating how to train very deep Transformers by removing shortcut connections (residual paths), which typically cause rank collapse in attention layers. The work has implications for understanding how information propagates through depth in Transformer architectures.

NLP Demystified

NLP Demystified is a free video course covering NLP fundamentals from text preprocessing through transformers and modern language models — aimed at practitioners who want a solid conceptual foundation rather than just API usage. One of the cleaner free NLP curricula available.

Training Language Models with Natural Language Feedback

Proposes learning from natural language feedback on model outputs rather than simple comparison labels, using a generate-filter-finetune loop to train GPT-3 to human-level summarization with only 100 feedback samples. Natural language carries more alignment signal per human evaluation than pairwise comparisons.

Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

Comprehensive survey by Crothers, Japkowicz, and Viktor mapping the threat models and detection methods for machine-generated text, with emphasis on fairness and robustness. As generative models like ChatGPT become widely accessible, the gap between reliable detection and adversarial evasion has become one of the central unsolved problems in AI safety and content integrity.

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

The BLOOM paper introducing a 176-billion parameter open-access multilingual language model trained by the BigScience collaborative on 46 natural languages and 13 programming languages. It demonstrated that a community-organized research effort could produce a frontier-scale LLM without proprietary infrastructure.

Rhasspy — Offline Private Voice Assistant

Rhasspy is a fully offline, privacy-first voice assistant framework supporting many human languages — no cloud required. Designed for Home Assistant integration and runs entirely on local hardware like a Raspberry Pi.

TART: Task-Aware Retrieval with Instructions

TART (Task-Aware Retrieval with Instructions) introduces BERRI, a dataset of ~40 retrieval tasks annotated with human-written task instructions, and trains a multi-task retrieval system that adapts its behavior based on explicit instructions. TART outperforms much larger models on BEIR by understanding the user's intent rather than just matching query-document similarity.

DIFFUSER: Discrete Diffusion via Edit-Based Reconstruction

Introduces DIFFUSER, an edit-based text generation model that adapts denoising diffusion to discrete text by framing generation as iterative editing rather than left-to-right token production. Competitive with autoregressive models on translation and summarization while enabling unique capabilities like prototype-conditioned generation and iterative revision.

Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs

Sap et al. test LLMs on theory-of-mind tasks and find they rely on spurious correlations and dataset artifacts rather than genuine social reasoning. Models perform well on training distributions but fail on adversarial or decontextualized ToM tests — a gap that matters for social AI applications.

Mapping Language Models to Grounded Representations

An NLP paper from late 2022 on mapping language model representations to grounded meaning — likely examining whether and how LLM internal representations correspond to real-world referents or structured semantic formalisms. Addresses the 'grounding problem' at the heart of debates about whether LLMs understand language.

ROOTS Search Tool — BigScience

A Hugging Face Space for searching ROOTS — the massive multilingual dataset used to train BLOOM, the BigScience open LLM. Lets researchers trace which training documents a model might have learned from.

A Complete Introduction to Prompt Engineering

Mihail Eric's comprehensive introduction to prompt engineering for LLMs — covering few-shot prompting, chain-of-thought, instruction tuning, and evaluation. Published in late 2022 when prompt engineering was emerging as a recognized discipline.

Wide Attention Is The Way Forward for Transformers

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

On the Paradox of Learning to Reason from Data

Zhang, Li, Meng, Chang, and Van den Broeck (UCLA) show that BERT achieves near-perfect accuracy on in-distribution logical reasoning problems while completely failing to generalize to other distributions over the same problem space. The explanation: BERT learned statistical features of the logical reasoning distribution, not the underlying reasoning function — a fundamental distinction between benchmark performance and genuine reasoning.

Attention? Attention! (Lilian Weng)

Lilian Weng's canonical blog post on attention mechanisms — written in 2018, it covers sequence-to-sequence attention, self-attention, and multi-head attention. Still the clearest single-page reference for understanding how attention works before reading the Transformer paper.

AI Content Generation, Part 1: Machine Learning Basics (Jon Stokes)

Jon Stokes' accessible introduction to machine learning as the foundation of AI content generation — Part 1 of a series aimed at readers with no ML background. Stokes covers the statistical learning framing without requiring math, making it one of the better on-ramps for non-technical audiences.

Looria: Reddit-Powered Product Research

Looria analyzes Reddit posts and comments to surface the most-recommended products in any category — turning unstructured community sentiment into structured product rankings. A practical application of NLP on social data to solve a real consumer decision problem.

A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems

A BERT-based dialog state tracking model optimized for resource-limited systems — achieving competitive performance on MultiWOZ while reducing model size and inference cost. Demonstrates that careful architecture choices can close the gap between full-scale models and edge-deployable alternatives.

"Hey Machine, What's My Invoice Total?" (Impira Blog)

Impira's explainer on document understanding ML — how to extract structured data like invoice totals from unstructured documents using layout-aware models. Covers why document AI is harder than it looks and how layout-aware transformers like LayoutLM changed the problem.

The Cost of Training NLP Models: A Concise Overview

AI21 Labs' 2020 overview of the financial and compute cost of training large NLP models, tracing the exponential growth in training expenditure from BERT through GPT-3. An early quantitative lens on the economics of scale in language model development.

PEER: A Collaborative Language Model

PEER is a language model trained to collaboratively write text with humans — drafting, suggesting edits, following instructions, and explaining its changes. An early formalization of human-AI collaborative writing that anticipates modern writing assistant workflows.

Modern NLP with Large Language Models (Sinan Ozdemir, Maven)

Sinan Ozdemir's Maven cohort course on modern NLP with GPT-3/4 and BERT — covering information retrieval, multi-task pipelines, and prompt engineering. One of the early structured courses teaching practitioners how to build with LLMs.

Google Talk to Books

Google's Talk to Books lets you search a large corpus of books using natural language statements, returning passages that respond semantically to your query. An early public demonstration of semantic search over a curated corpus, predating the vector database era.

The Annotated Transformer

The Annotated Transformer walks through the 'Attention Is All You Need' paper with working PyTorch code alongside every equation — the canonical resource for understanding transformer architecture from first principles. Published by Harvard NLP.

AlexaTM 20B: Amazon's Few-Shot Language Model

AlexaTM 20B is Amazon's 20B parameter seq2seq language model that outperforms PaLM 540B on one-shot summarization and sets state-of-the-art on multilingual translation — while training on one-fifth of GPT-3's carbon footprint. A strong argument for encoder-decoder architecture in few-shot settings.

SLED: Efficient Long-Text Understanding with Short-Text Models

SLED (Sliding-Encoder and Decoder) lets you apply short-context pretrained models to arbitrarily long documents by chunking input with overlap and fusing representations in the decoder. Competitive with specialized long-context models without the expensive custom pretraining.

Re:infer — Conversational Data Analytics

Re:infer is an enterprise platform for analyzing and automating unstructured conversational data — emails, chats, tickets — using NLP. Acquired by UiPath in 2022, it's a representative example of the NLP-for-customer-operations category that emerged as transformers made text classification production-ready.

Differentiable Prompt Makes Pre-trained Language Models Better Few-Shot Learners

DifferentiablePrompt (DPT) replaces discrete token prompts with optimized continuous embeddings, enabling gradient-based prompt tuning for few-shot learning. Published at ICLR 2022, it established that soft prompts can match full fine-tuning with far fewer parameters.

DART: Differentiable Prompt Makes Pre-Trained Language Models Better Few-Shot Learners

DART (Differentiable pRompT) trains prompt templates end-to-end via backpropagation, treating prompts as learnable continuous vectors rather than fixed text. It makes small pre-trained language models competitive few-shot learners without scaling to GPT-3 sizes — an important stepping stone between hand-crafted prompts and full fine-tuning.

Pretrain, Prompt, Predict: NLP Survey

A comprehensive survey of the Pretrain, Prompt, Predict paradigm in NLP — covering prompt engineering, answer engineering, and in-context learning as a unified framework. The academic backbone behind what practitioners call 'prompt engineering'.

Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Reynolds and McDonell (2021) argue that prompting large language models is better understood as programming than as few-shot learning — the few examples in a prompt aren't training data but rather code that specifies the desired computation. This reframing opens up principled prompt design strategies that beat naive templating.

SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

Vu, Constant, Al-Rfou, Cer, and Lester (Google Research/UMass, 2022) show that initializing soft prompts from a related source task dramatically improves prompt tuning, achieving near fine-tuning performance while keeping the base model frozen. The result reveals that prompt initialization is a critical and underappreciated factor in parameter-efficient adaptation.

Transformer Models: An Introduction and Catalog

Xavier Amatriain's catalog of transformer models covers the full landscape of architecture variants and named models as of mid-2022 — encoder-only, decoder-only, encoder-decoder, and multimodal. A useful orientation map for the pre-ChatGPT era of rapid model proliferation.

Pre-Trained Models: Past, Present and Future

Comprehensive survey of large-scale pre-trained models (PTMs) tracing the evolution from BERT and GPT through four research frontiers: architecture, contextual use, efficiency, and interpretability. Required reading for understanding how self-supervised pre-training became the unified backbone of modern AI.

Refinery: Open-Source Data-Centric IDE for NLP

Refinery is an open-source data-centric IDE for NLP — combining programmatic labeling, data management, and neural search in one tool. The code-kern-ai take on Snorkel's programmatic labeling approach, extended with neural search for exploration.

Jay Alammar — Visualizing Machine Learning

Jay Alammar's blog is the go-to resource for visually understanding modern ML architectures — transformers, BERT, GPT, and more through hand-crafted diagrams. His illustrated explainers have become canonical references for practitioners who want intuition before equations.

BLOOM Training Corpus: 2D Embedding Visualization

A 2D UMAP visualization of 10 million text chunks from the BLOOM training corpus, encoded with all-distilroberta-v1. A rare window into the geometry of a frontier model's pretraining data.

Text Embeddings Visually Explained

Cohere's visual primer on text embeddings explains how words and sentences become vectors in high-dimensional space, and what operations on those vectors mean semantically. A good conceptual foundation before diving into practical embedding-based applications like semantic search or classification.

ART: Questions Are All You Need to Train a Dense Passage Retriever

ART (Autoencoding-based Retriever Training) trains dense retrievers without labeled question-passage pairs — only questions and an unpaired document collection. This removes the main bottleneck for deploying dense retrieval in new domains where annotation is expensive.

PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

PromptSource is an IDE and community repository for creating, sharing, and iterating on natural language prompts that map dataset examples to input-output pairs for language model training and evaluation. With over 2,000 prompts for ~170 datasets, it provided the infrastructure behind the T0 family of models and multitask prompted training research.

Unifying Language Learning Paradigms (UL2)

Google Research's UL2 paper proposes Mixture-of-Denoisers (MoD), a unified pre-training objective that combines span corruption, prefix LM, and causal LM into a single framework — and separates architecture choices from pre-training objectives, which were previously conflated. The insight that objective and architecture are orthogonal opened the door to mixing paradigms that were previously treated as distinct camps.

Gradient Explanations for HuggingFace BERT Classification

A tutorial by Victor Dibia on generating gradient-based explanations for HuggingFace BERT text classification models in TensorFlow 2.0 — visualizing which tokens most influenced the model's prediction. Explainability for transformer classifiers was a practical gap in 2022 since attention maps alone are insufficient.

Modeling Vocabulary for Big Code Machine Learning

An empirical study of vocabulary modeling decisions for machine learning systems on source code, evaluated across 14,436 projects. It matters because the choices made when tokenizing and preprocessing code vocabularies have an outsized impact on neural language model accuracy, yet were poorly documented before this work.

BERTopic: The Future of Topic Modeling

A Pinecone explainer on BERTopic — a topic modeling library that uses transformer embeddings instead of bag-of-words statistics, producing semantically coherent topics. BERTopic largely made LDA obsolete for practitioners who have access to modern embeddings.

Ranking YC Companies with a Neural Net

Eric Jang trains a neural net to rank Y Combinator companies by prestige/success using only their names and descriptions — an experiment in whether language models encode meaningful startup quality signals. The results are surprisingly coherent.

Haystack: Question Answering at Scale

Haystack by deepset is an open-source NLP framework for building question-answering and search systems at scale — Retriever-Reader pipelines over large document corpora. An early entrant in what became the RAG ecosystem.

Ntropy: Transaction Enrichment API

Ntropy is a transaction enrichment API that cleans, labels, and categorizes raw banking transaction data using ML — turning the noisy strings in bank feeds into structured, queryable merchant and category data.

Word2Vec Explained

A clear walkthrough of Word2Vec's intuition and mechanics — skip-gram vs. CBOW, negative sampling, and why the king-queen analogy works. Good primer before reading papers on later embedding methods.

AI Index Report 2021

Stanford HAI's fourth annual AI Index Report tracking AI research output, compute trends, investment, and societal impact through 2020. A useful baseline document for understanding how rapidly the field accelerated in the years immediately following.

Document Processing in Multiple Languages

Nanonets' overview of multilingual document processing — extracting structured data from invoices, forms, and documents in non-English languages. Covers the ML challenges of multilingual OCR and key-value extraction at scale.

Task-Specific Knowledge Distillation for BERT

A tutorial on task-specific knowledge distillation for BERT using Hugging Face Transformers and Amazon SageMaker — compressing a 109M-parameter BERT-base into a 4M-parameter student with 90%+ performance retention. Demonstrates that you don't need a massive model in production if you can distill task knowledge from one.

ML and NLP Research Highlights of 2021

Sebastian Ruder's annual ML/NLP research highlights for 2021 — covering foundation models, the prompting revolution, AlphaFold 2, diffusion models, and the growing focus on efficiency and responsible AI. The most useful single-document summary of where the field moved that year.

Financial Text Summarization with Hugging Face and Keras

A tutorial on fine-tuning distilled BART for financial news summarization using Hugging Face Transformers with Keras and Amazon SageMaker — generating headline-length summaries from longer articles. A practical demonstration of seq2seq fine-tuning on domain-specific data.

How Sber Built ruDALL-E — Interview with Sergei Markov

Serokell's interview with Sergei Markov of SberDevices about building ruDALL-E — a 12B parameter Russian-language text-to-image model. Covers the engineering and research challenges of training massive multimodal models, plus the open-source culture argument in ML.

Sentence Transformers: Pretrained Models

The Sentence-BERT pretrained models documentation — a reference for choosing the right sentence embedding model for semantic similarity, semantic search, or paraphrase detection tasks. The go-to resource when you need to pick a model before training your own.

Semantic Textual Similarity — Papers with Code Benchmark

Papers with Code's benchmark page for Semantic Textual Similarity — a task measuring how similar two sentences are in meaning, scored against human judgments. The canonical reference for comparing embedding models on this fundamental NLP task.

jupyter-text2code: English to Python in Jupyter

jupyter-text2code is a proof-of-concept Jupyter extension that converts English queries into Python code — an early 2021 demonstration of natural language to code generation in notebooks, before Copilot and ChatGPT made this mainstream.

Berkeley Document Summarizer

The Berkeley Document Summarizer is a learning-based extractive summarization system that uses syntactic compression and coreference constraints. Academic research code from Greg Durrett, representing the pre-neural era of NLP summarization work.

An Introduction to Knowledge Graphs

Stanford AI Lab's introduction to knowledge graphs — what they are, how they're constructed, and where they're used. A solid conceptual overview covering entity linking, relation extraction, and the gap between structured and unstructured knowledge.

GPT-Neo: Open Source GPT-3 Scale Language Models

GPT-Neo is EleutherAI's open-source implementation of GPT-style language models at GPT-3 scale — the first serious open attempt to replicate GPT-3's capabilities before open-weight models became common. Historically significant as the origin of the open LLM movement.

skift: scikit-learn Wrappers for fastText

skift wraps Facebook's fastText text classifiers in scikit-learn's estimator API, making fastText accessible as a drop-in component in scikit-learn pipelines and GridSearchCV. Useful for fast, production-grade text classification without leaving the sklearn ecosystem.

150+ Best Machine Learning, NLP, and Python Tutorials

Robbie Allen's curated list of 150+ machine learning, NLP, and Python tutorials, organized by topic and difficulty. A high-signal link aggregation for 2020-era ML self-study, with commentary on what each resource covers.

Speech and Language Processing (Jurafsky & Martin)

The free online draft of Jurafsky and Martin's 'Speech and Language Processing' — the canonical NLP textbook, covering everything from regex to transformers, now in its 3rd edition. Freely available online in draft form and unusually well-written for a technical textbook.

Over 150 of the Best Machine Learning, NLP, and Python Tutorials

A 2017 curated list of 150+ ML, NLP, and Python tutorials organized by topic — a snapshot of the best practitioner learning resources from the pre-transformer era. Useful as a historical reference for what the field considered canonical learning material at the time.

LexVec

LexVec is a Go implementation of a word embedding model that factorizes the PPMI matrix with position-dependent weighting, outperforming Word2Vec on several NLP benchmarks at the time. An interesting mid-2010s data point in the race to improve on Word2Vec before attention-based models took over.

Voynich Manuscript: Word Vectors and t-SNE Visualization

Christian Perone applies word2vec embeddings and t-SNE visualization to the Voynich Manuscript — an undeciphered 15th-century text — to surface structural patterns in its unknown script. A creative application of NLP tools to a centuries-old mystery.

Topic Modeling of Twitter Followers

A tutorial applying LDA topic modeling to Twitter follower descriptions to segment an audience by interest — one of the cleaner early examples of using unsupervised NLP to understand social media audiences programmatically.

What is TF-IDF? The 10 Minute Guide

A 10-minute introduction to TF-IDF — the classic term-weighting scheme that balances how often a word appears in a document against how rare it is across the corpus. Still one of the most useful baselines in text analysis despite being decades old.

Building the Next New York Times Recommendation Engine

The NYT engineering blog's post on building their new recommendation engine — combining collaborative filtering with content-based signals to recommend articles. A rare look at production recommendation systems at a major media company before the algorithmic feed era fully arrived.

Parsing English with 500 Lines of Python

Matthew Honnibal's post describing a fast dependency parser for English implemented in 500 lines of Python — a precursor to spaCy. Demonstrates that a useful NLP system doesn't need a massive codebase if the algorithm is right.

LinkedIn Acquires Bright: Big Data Job Matching

LinkedIn's ~$120M acquisition of Bright, a job-matching startup that used machine learning to score candidate-job fit from resume and job description text. A signal of how ML-powered matching was becoming central to professional network platforms.

Stanford CoreNLP: Free Text Analysis Pipeline

Stanford's CoreNLP — the Java-based NLP pipeline from Stanford's NLP Group offering tokenization, POS tagging, NER, parsing, coreference resolution, and sentiment analysis in one tool. The dominant academic NLP pipeline before deep learning NLP models took over.

Troll Detection with Scikit-Learn

Impermium's Kaggle blog post on building a troll detection classifier with scikit-learn — text features, gradient boosting, and the practical challenges of training on imbalanced toxic comment data.

Teaching a Computer to Read: NLP Hacking in Python

Scripted blog's introduction to NLP in Python — using NLTK for tokenization, part-of-speech tagging, named entity recognition, and sentiment analysis. A practical hands-on introduction to text processing written for a content-tech company's engineering blog.

Machine Learning Tutorial: The Naive Bayes Text Classifier

DatumBox's tutorial on the Naive Bayes text classifier — explaining the math, the conditional independence assumption, and how to implement it from scratch. A standard first classifier for anyone learning NLP and ML simultaneously.

Naive Bayes Classification in SQL at Zipfian Academy

A Zipfian Academy student's tweet about implementing Naive Bayes classification in SQL, categorizing the New York Times corpus one word at a time. A striking demonstration that ML concepts can be expressed in unexpected substrate.

Teaching a Computer to Read: NLP Hacking in Python

Scripted's NLP hacking tutorial in Python — covering tokenization, part-of-speech tagging, named entity recognition, and text classification with NLTK and scikit-learn. An applied introduction to NLP for data scientists.

Recursive Deep Models for Semantic Compositionality

Stanford's Recursive Neural Tensor Network paper and Sentiment Treebank dataset — Richard Socher's EMNLP 2013 work that used tree-structured recursive neural networks to predict fine-grained sentiment at every node of a parse tree. A landmark paper that pushed NLP models toward compositionality.

Natural Language Processing with Apache Hadoop and Python

Cloudera's 2010 post (bookmarked in 2013) on running natural language processing pipelines with Apache Hadoop and Python using Hadoop Streaming — an early recipe for scaling NLP beyond single-machine limits using commodity clusters.

Many Downloadable Twitter Archives Available for Researchers

DataScholars on downloadable Twitter archives available for academic researchers — a 2013 directory of public Twitter datasets for NLP, social network analysis, and computational social science. A snapshot of open Twitter data before the API became restrictive.

Natural Language Processing for the Working Programmer

Natural Language Processing for the Working Programmer — a free online book teaching NLP concepts using Haskell. Unusual choice of language for NLP education; valuable for functional programming practitioners interested in text processing.

Hidden Markov Models on Hadoop — Isabel Drost

Isabel Drost's slides on Hidden Markov Models and Hadoop — covering how HMMs can be implemented and trained at scale using MapReduce. A 2012-era reference for scaling sequence models before deep learning displaced them.