Subject
13 entries
Streaming
Bookmarks
Estuary Flow: Apache Iceberg Materialization Connector
Estuary Flow's Apache Iceberg materialization connector writes streaming Flow collections into Iceberg tables on object storage — bridging real-time CDC streams into the open lakehouse table format. A key integration for streaming-to-lakehouse pipelines.
VDO.Ninja: Free P2P Live Video via WebRTC
VDO.Ninja (formerly OBS.Ninja) is a free, no-account P2P video streaming tool that brings live video from phones, cameras, and remote guests directly into OBS Studio or any browser via WebRTC. The simplest way to pull in a remote camera feed without third-party software or monthly fees.
Gazette Core: Unified SQL, Batch, and Stream Processing
Gazette is an open-source platform for building data pipelines that mix SQL, batch, and stream processing in a unified model backed by journals (append-only logs). Designed for organizations that want Kafka-like durability with flexible processing paradigms without committing to a full streaming architecture.
Algorithms and Data Structures for Massive Datasets
Manning 2021 textbook covering algorithms and data structures built for massive datasets — Bloom filters, HyperLogLog, Count-Min Sketch, LSH, and streaming algorithms. Practical treatment of how to handle data that won't fit in memory or where exact answers are too expensive.
How I VTuber — Xe Iaso's Setup
Xe Iaso's technical walkthrough of their VTubing setup — the software, avatar rigging, face tracking, and streaming stack used to stream as a virtual character while keeping a developer's privacy. Practical guide at the intersection of live coding and virtual avatar streaming.
Real-Time Machine Learning: Challenges and Solutions
Chip Huyen's definitive survey of real-time ML challenges — the engineering gap between batch ML pipelines and systems that must make predictions on live data with low latency. Covers online prediction, continual learning, and the infrastructure needed to bridge training and serving.
Yelp Pyleus — Apache Storm Topologies in Pure Python
Yelp open-sources Pyleus — a framework for writing Apache Storm stream processing topologies in pure Python. Solved the JVM barrier that kept Python data engineers from using Storm's real-time streaming capabilities.
Spark, Storm and Real-Time Analytics
An early comparison of Apache Storm and Spark Streaming for real-time analytics — covering their different processing models, latency characteristics, and use cases. A snapshot of the stream processing options available before Kafka Streams, Flink, and other tools matured.
Timely Dataflow: An Introduction
An introduction to Timely Dataflow, the distributed computation model developed at Microsoft Research that unified batch and streaming processing through a novel timestamp-based progress tracking system. A technically significant but underappreciated alternative to the Spark/Storm paradigm.
Streaming MapReduce with Summingbird
Twitter's Summingbird library unifies batch (Hadoop/Scalding) and streaming (Storm) MapReduce into a single Scala API — write once, run on either backend. An early practical implementation of the Lambda Architecture's dual-speed compute model.
Interactive Big Data Analysis Using Approximate Answers
O'Reilly Strata coverage of approximate query processing for interactive big data analysis — using sketches and sampling to get fast approximate answers over large datasets when exact computation is too slow. An important design pattern for data products that prioritize responsiveness over precision.
Introduction to Apache Kafka (TriHUG, July 2013)
TriHUG July 2013 talk introducing Apache Kafka — the distributed log system LinkedIn built and open-sourced. Caught at the moment Kafka was still an unfamiliar tool to most data engineers, before it became the de-facto streaming backbone of the modern data stack.
Tweetping — Real-Time Twitter Visualization
Tweetping visualized the Twitter streaming API as a real-time pulsing global map — each tweet a dot appearing at its geolocation. A landmark example of combining streaming data APIs with real-time browser visualization.