Subject
13 entries
Observability
Bookmarks
Agent Super Spy: LLM observability for local agent development
Local LLM proxy stack that gives you full visibility into what your AI agents actually do. Dual-layer observability via OpenTelemetry traces and raw HTTP capture.
Mnemom: proving what AI agents are thinking
Mnemom.ai makes AI agent reasoning transparent — "Prove What Your AI Agents Are Thinking." An agent explainability tool for validating agent decision-making.
tapes: transparent AI agent telemetry
Tapes introduces transparent telemetry for AI agents — a structured trace format that makes agent execution auditable without requiring a full observability platform. Good complement to heavier tools like Arize AX or Phoenix.
Context Lens: see what's filling your context window
Context Lens is a local LLM API proxy that shows what's filling your context window — system prompts vs. tool definitions vs. conversation history vs. tool results. Answers the question every developer asks: why is this session so expensive?
Plano: AI-native proxy and data plane for agents
Plano is an AI-native proxy and data plane for agentic apps — centralized routing, guardrails, observability, and LLM switching so you don't re-implement plumbing in every codebase. Decouples agents from brittle framework abstractions.
HyperContext: spatial context visualization for AI sessions
HyperContext visualizes AI session state as a spatial map — see what the model actually knows, how the context window is structured, and where attention is distributed. Makes the invisible context window visible.
Sherlock: real-time LLM API traffic interceptor and dashboard
Sherlock intercepts LLM API traffic and shows token usage, costs, and context window consumption in a real-time terminal dashboard. Debug prompts and track costs across AI development sessions without adding SDK instrumentation.
A Practitioner's Guide to Wide Events
Jeremy Morrell's practitioner guide to wide events in observability — high-cardinality single-row records that capture the full context of a request in one place, rather than scattered low-cardinality metrics and fragmented logs. Covers implementation details that other wide-events explainers skip.
TensorZero
TensorZero is a self-hosted LLMOps platform that unifies gateway, observability, evaluation, optimization, and A/B testing in one Rust-built system. The feedback loop: production inference data flows back into prompt and model improvement automatically.
Braintrust Data: LLM Evaluation Platform
Braintrust is an AI evaluation platform for logging, scoring, and comparing LLM outputs across experiments and prompt versions. Addresses the core LLMOps problem: how do you know if your prompt change made things better or worse?
Autoblocks: Debug and Monitor Generative AI Features
Autoblocks is a developer-centric monitoring and debugging platform for LLM features — trace LLM calls, replay sessions, catch regressions, and improve model behavior with production data. Part of the 2023 LLMOps tooling wave.
All the Hard Stuff Nobody Talks About When Building with LLMs
Honeycomb's post-mortem on building their LLM-powered Query Assistant — the engineering challenges they didn't expect, including output validation, latency at the tail, prompt brittleness, and user trust. Unusually honest practitioner account from a team that shipped an LLM feature to production.
Aporia — ML Observability
Aporia is a cloud-native ML observability platform for monitoring deployed models — drift detection, performance monitoring, and explainability tooling. Addresses the gap between training-time metrics and what models actually do in production.