Reasoning

LLM Council: multi-model deliberation as a Claude Code plugin

LLM Council is a Claude Code plugin that runs a 3-phase multi-model deliberation on any query: parallel responses, cross-model ranking, chairman synthesis. Built by DAIR.AI on Fireworks AI inference.

The Future Of Reasoning

A YouTube video titled 'The Future Of Reasoning' — content unavailable for fetching, but likely covers the state and trajectory of machine reasoning capabilities in AI systems.

Use Prolog to Improve LLM Reasoning

A proposal for using Prolog as a symbolic reasoning layer alongside LLMs — offloading formal logical inference to Prolog while the LLM handles natural language parsing and generation. A neuro-symbolic hybrid that addresses LLMs' known weakness at strict logical deduction.

DSPy Self-Discover Framework

A DSPy + Groq-powered framework that takes natural language task descriptions and generates executable Python code to solve them — automated problem-solving through AI-driven reasoning and code generation. Now archived.

Graph of Thoughts: Solving Elaborate Problems with LLMs

Graph of Thoughts (GoT) extends chain-of-thought and tree-of-thought prompting to arbitrary graph structures — letting LLM reasoning steps combine, branch, and loop rather than just proceeding linearly. More expressive than CoT or ToT for complex multi-step problems.

Reasoning or Reciting? LLM Capabilities via Counterfactual Tasks

An arxiv paper testing whether LLMs are reasoning or reciting — using counterfactual tasks (modified versions of standard benchmarks) to distinguish genuine reasoning from pattern-matching on training data. Finds LLMs show sharp performance drops on counterfactual variants, suggesting heavy reliance on memorization.

llm-reasoners: Advanced LLM Reasoning Algorithms

llm-reasoners is a library for advanced LLM reasoning algorithms — implementing Tree of Thoughts, RAP (Reasoning via Planning), and other structured reasoning approaches over standard chain-of-thought. Useful for research into how to get LLMs to reason more reliably on complex tasks.

Self-Refine: Iterative LLM Output Improvement via Self-Feedback

Self-Refine is a framework where LLMs generate feedback on their own outputs and iteratively refine them — no human feedback, no gradient updates. Shows that a single LLM can be its own critic and improve outputs across diverse tasks.

Program of Thoughts Prompting: Disentangling Computation from Reasoning

Program of Thoughts separates reasoning from computation by having LLMs write executable Python programs rather than performing arithmetic inline, delegating number-crunching to an interpreter. It substantially outperforms chain-of-thought on numerical reasoning benchmarks by eliminating the arithmetic errors that plague prose reasoning chains.

PAL: Program-aided Language Models

PAL prompts LLMs to generate Python programs as intermediate reasoning steps, then delegates actual computation to an interpreter, outperforming chain-of-thought on math and symbolic tasks. It neatly separates what LLMs are good at (translating problems to code) from what they're bad at (arithmetic).

STaR: Bootstrapping Reasoning With Reasoning

Large Language Models Can Self-Improve

Huang et al. (2022) show that LLMs can bootstrap their own reasoning ability by generating chain-of-thought rationales, filtering with self-consistency majority vote, and finetuning on the high-confidence outputs — no human labels needed. A clean demonstration that LLMs can improve themselves without supervised signal.

On the Paradox of Learning to Reason from Data

Zhang, Li, Meng, Chang, and Van den Broeck (UCLA) show that BERT achieves near-perfect accuracy on in-distribution logical reasoning problems while completely failing to generalize to other distributions over the same problem space. The explanation: BERT learned statistical features of the logical reasoning distribution, not the underlying reasoning function — a fundamental distinction between benchmark performance and genuine reasoning.

Solving Quantitative Reasoning Problems with Language Models (Minerva)

Lewkowycz et al. at Google Research introduce Minerva, a language model pretrained on general text and further trained on technical content that achieves state-of-the-art on quantitative reasoning benchmarks without external tools. It correctly answers nearly a third of undergraduate-level science problems — an early proof that domain-specific pretraining unlocks STEM reasoning at scale.