Subject
56 entries
Data Engineering
Bookmarks
Estuary Flow: Apache Iceberg Materialization Connector
Estuary Flow's Apache Iceberg materialization connector writes streaming Flow collections into Iceberg tables on object storage — bridging real-time CDC streams into the open lakehouse table format. A key integration for streaming-to-lakehouse pipelines.
Dequery: The SQL Debugger
Dequery is a SQL debugger that traces data lineage step-by-step through CTEs, subqueries, JOINs, and window functions — showing which source rows contributed to each result. Fills the long-standing gap between SQL query execution and traditional debugging.
Superlinked
Superlinked is a framework for data engineers to build vector embeddings from structured data with control over how different attributes are weighted and combined. Addresses the gap between raw ML embedding models and production multi-attribute vector search.
Data Engineering Design Patterns (DEDP)
Data Engineering Design Patterns (DEDP) is a free online book covering canonical patterns for building data pipelines — ingestion, transformation, storage, and orchestration. Structured as a pattern catalog rather than a tutorial, useful as a reference for recurring architectural decisions.
Gazette Core: Unified SQL, Batch, and Stream Processing
Gazette is an open-source platform for building data pipelines that mix SQL, batch, and stream processing in a unified model backed by journals (append-only logs). Designed for organizations that want Kafka-like durability with flexible processing paradigms without committing to a full streaming architecture.
ethereum-etl: Python ETL for Ethereum Blockchain Data
ethereum-etl is the standard Python library for extracting Ethereum blockchain data into structured formats — blocks, transactions, ERC-20 transfers, receipts, logs, internal transactions — with Google BigQuery export support. The reference implementation for blockchain data pipelines.
Building Blockchain Data Pipelines with The Graph & Singer
A tutorial on building blockchain data pipelines using The Graph's subgraph queries piped through Singer, the open-source ETL standard. Shows how to treat on-chain indexed data like any other data source in a standard analytics stack.
Apollo: Cross-Chain ETL for EVM Chaindata
Apollo is a cross-chain ETL tool from Chainbound for extracting raw EVM chaindata — transactions, logs, traces — across multiple networks simultaneously. Designed for analysts who need low-level on-chain data without running their own full archive nodes.
Designing Cloud Data Platforms
Manning textbook on building modern cloud data platforms covering ingestion, storage, processing, and serving layers across major providers. Practical guide for data engineers designing end-to-end pipelines that balance performance, cost, and operational complexity.
Data Mesh: Delivering Data-Driven Value at Scale
Zhamak Dehghani's 2022 O'Reilly book defines data mesh — a sociotechnical approach to data architecture that treats data as a product owned by domain teams, distributed across a federated data platform, and governed by global standards without centralized control. The book is the canonical reference for moving beyond monolithic data lakes and warehouses.
Ponder: Pandas at Scale
Ponder is a startup that makes Pandas run at scale without rewriting your code — a drop-in compatibility layer that runs standard Pandas operations on distributed backends. Targets the massive installed base of data scientists who know Pandas but hit its single-machine limits.
MLOps Notebooks Syllabus
A Jupyter notebook curriculum for MLOps — covering model deployment, monitoring, CI/CD for ML, feature stores, and data versioning. Practical operational coverage for the gap between training a model and running it reliably in production.
Free Data Engineering Learning Resources
Pipeline Data Engineering Academy's curated list of free data engineering learning resources — covering SQL, Python, Spark, Airflow, dbt, and cloud data platforms. A structured entry point for engineers transitioning into data engineering roles.
The Unbundling of Airflow
fal.ai's analysis of why Apache Airflow is being unbundled by specialized tools — Prefect and Dagster for orchestration, dbt for transformation, Temporal for long-running workflows. The monolithic DAG orchestrator is fracturing along functional lines.
PipelineDP: Differentially Private Data Aggregation
PipelineDP is an open-source framework from Google and OpenMined for differentially private data aggregation at scale — extract insights from large datasets while provably protecting individual privacy. Brings differential privacy out of academia and into data pipeline tooling.
Data Engineering Zoomcamp
DataTalksClub's free 9-week data engineering course — builds an end-to-end pipeline covering Docker, Terraform, BigQuery, dbt, Spark, and Kafka. One of the better free paths from zero to production data engineering.
Lessons on ML Platforms from Netflix, DoorDash, Spotify, and More
Ernest Chan's Towards Data Science survey of ML platform design at Netflix, DoorDash, Spotify, and other tech companies — extracting common patterns and lessons from how production ML infrastructure evolved at scale. Practical systems thinking for ML platform builders.
Feature Stores — A Hierarchy of Needs
Eugene Yan's survey of feature stores organized as a hierarchy of needs — from basic feature reuse to online serving to real-time streaming. Practical taxonomy for understanding when you need a feature store and what tier of sophistication your use case requires.
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh
Zhamak Dehghani's foundational essay (hosted on martinfowler.com) proposing data mesh — treating data as a product owned by domain teams rather than centralized in a monolithic data lake. The conceptual origin of the data mesh movement.
How Affirm Uses AWS Fargate and Apache Airflow for Batch Jobs
Affirm's engineering blog post on running Apache Airflow DAGs on AWS Fargate for batch job management — serverless task execution without persistent workers. A practical architecture case study for ML and data pipeline orchestration at a fintech scale.
Emerging Architectures for Modern Data Infrastructure
Andreessen Horowitz maps the emerging stack for modern data infrastructure — separating analytics, ML, and operational workloads into distinct architectural patterns. A useful taxonomy of the data tooling landscape circa 2020.
Building a Data Lake with AWS S3 and Athena
A tutorial on replacing a costly Redshift cluster with a serverless data lake using AWS S3 and Athena — demonstrating 16-minute queries reduced to 11 seconds at 10 cents a run using Parquet, partitioning, and Glue ETL. Practical architectural migration case study.
Kickstarter SQL Style Guide
Kickstarter's SQL style guide — conventions for formatting, naming, CTEs, joins, and documentation that prioritize reproducibility and shareability across a data team. A practical reference for writing SQL others can read and maintain.
Data Science Meets DevOps: MLOps with Jupyter, Git, and Kubernetes
Kubeflow's blog post on MLOps practices combining Jupyter notebooks, Git, and Kubernetes — the infrastructure side of making ML reproducible, collaborative, and deployable. Establishes the canonical MLOps stack for Kubernetes-based ML teams.
Airflow and XCom: Inter-Task Communication Use Cases
A guide to Airflow's XCom (cross-communication) mechanism for passing data between tasks in a DAG — covering when to use it, when to avoid it, and practical use cases. XCom is one of the most misused Airflow features.
We're All Using Airflow Wrong and How to Fix It
Bluecore Engineering's argument that most Airflow deployments misuse operators — using Python, Bash, and Spark operators that run code inside the Airflow worker, which breaks isolation and causes resource contention. The fix: use KubernetesPodOperator for everything.
Insight Data Engineering Ecosystem Map
Insight Data Science's map of the data engineering ecosystem circa 2017 — a diagram organizing dozens of tools by pipeline stage (ingest, store, process, visualize). A widely shared snapshot of the Hadoop/Spark era's explosion of competing infrastructure tools.
Is It Pokemon or Big Data?
A quiz that asks whether a name belongs to a Pokemon or a Big Data tool — hilariously difficult because the Hadoop ecosystem's naming conventions are indistinguishable from fictional creature names. A sharp cultural critique disguised as a game.
Keys to Understanding: Data Scientist vs. Data Engineer
Domino Data Lab slides on knowing when to hire a data scientist versus a data engineer — clarifying the distinct skills, responsibilities, and organizational needs each role fills. From the Data Popup Seattle conference, when this distinction was still being established.
Building Analytics at 500px
A first-person account of building 500px's analytics infrastructure from scratch — Amazon Redshift data warehouse, Luigi ETL, Periscope BI. The 20% evangelism rule and 'don't bake your own ETL' lesson make it one of the most practical early data engineering retrospectives.
Top Mistakes Developers Make When Using Python for Big Data Analytics
A practical rundown of the top Python performance mistakes for big data workloads — covering generator vs. list comprehension choices, pandas anti-patterns, and when to reach for NumPy. Still relevant since Python's core performance traps haven't changed.
Yelp Pyleus — Apache Storm Topologies in Pure Python
Yelp open-sources Pyleus — a framework for writing Apache Storm stream processing topologies in pure Python. Solved the JVM barrier that kept Python data engineers from using Storm's real-time streaming capabilities.
Scalable A/B Experiments at Pinterest
Pinterest's engineering blog on how they built scalable A/B testing infrastructure — covering experiment assignment, metric computation, and statistical significance at Pinterest's scale. A practitioner's account of the gap between textbook A/B testing and production experimentation.
TSAR — TimeSeries AggregatoR
TSAR (TimeSeries AggregatoR) — a tool for time series aggregation, saved via a Prismatic share link that is no longer functional. The URL itself is broken; content unrecoverable.
Zipfian Academy Launches New Fellowship and Data Engineering Programs
Data Science 101's coverage of Zipfian Academy launching its Data Fellowship and Data Engineering programs — secondary source reporting on the announcement, adding community context from the data science blog ecosystem.
Zipfian Academy Launches Data Fellowship and Data Engineering Immersive
Zipfian Academy's official announcement of the Data Fellowship and Data Engineering Immersive programs — expanding from a single data science cohort to parallel tracks covering both analysis and engineering. A milestone in the bootcamp's evolution.
Random Sampling from Very Large Files
Practical techniques for taking random samples from large files without loading them into memory — covering Unix tools (shuf, awk) and reservoir sampling. Essential for working with data too large for pandas to read in one shot.
Pig Not-So-Foreign Language: Paper Notes
Bugra Akyildiz's notes on the 'Pig Latin: A Not-So-Foreign Language for Data Processing' paper by Yahoo! Research — explaining how Pig Latin compiles high-level data flow operations to MapReduce jobs. Captures why Pig was a meaningful step up from raw MapReduce for ETL work.
How Companies Are Using Spark
Strata/O'Reilly coverage of how companies were adopting Apache Spark in 2013, early in the engine's rise to ubiquity. A snapshot of early enterprise Spark use cases before it displaced Hadoop MapReduce as the default.
Rolling Average in Hive
Brent Ozar's walkthrough of computing rolling averages in Hive — a problem that looks like a simple SQL query but requires window functions or self-joins in Hive's then-limited SQL dialect. A practical data engineering puzzle from the Hadoop era.
Interactive Big Data Analysis Using Approximate Answers
O'Reilly Strata coverage of approximate query processing for interactive big data analysis — using sketches and sampling to get fast approximate answers over large datasets when exact computation is too slow. An important design pattern for data products that prioritize responsiveness over precision.
Hadoop's Unsung Sweet Spot: Unstructured ETL
A LinkedIn discussion arguing that Hadoop's real sweet spot wasn't analytics but unstructured ETL — transforming messy, heterogeneous data into structured forms before loading into traditional data warehouses. A nuanced counterpoint to the 'Hadoop replaces SQL' narrative of 2013.
Set Operations in the Unix Shell
Peteris Krumin's catonmat guide to implementing set operations (union, intersection, difference, complement) using Unix command-line tools. A concise demonstration that shell pipelines can express set algebra without any code.
Don't MAWK AWK – the Fastest and Most Elegant Big Data Munging Language
Brendan O'Connor's defense of AWK as a fast, elegant big data munging language — arguing it beats Python for many common structured text processing tasks. A counterpoint to the idea that awk is an obsolete curiosity.
14 Big Data Startups You're Going To Be Hearing About
Business Insider's 2013 list of 14 big data startups to watch — a market snapshot of the venture-backed companies building the infrastructure, analytics, and tooling layers of the emerging big data stack. A time capsule of which companies the industry thought would matter.
A Visual Explanation of SQL Joins
Jeff Atwood's Coding Horror post using Venn diagrams to explain SQL JOIN types visually. One of the most-referenced SQL explanations on the web — the mental model that finally makes INNER, LEFT, RIGHT, and FULL OUTER joins click.
A Programmer's Guide to Big Data: 12 Tools to Know
GigaOM's 2012 reference guide to 12 big data tools a programmer should know — covering Hadoop, Pig, Hive, HBase, Storm, and emerging alternatives. A snapshot of the Hadoop ecosystem at its peak complexity, before Spark simplified much of it.
Shark: Real-time Queries and Analytics for Big Data
O'Reilly Strata article on Shark — the precursor to Spark SQL that brought real-time interactive queries to Hadoop/Spark in 2012. Part of the wave of tools (Impala, Shark, Drill) that challenged Hive's batch-query dominance.
HDFS Has Won: De Facto Standard for Centralized Data Storage
A 2012 claim that HDFS had emerged as the de facto standard for centralized big data storage — a snapshot of a moment when Hadoop's dominance seemed settled. Written just before the data lake era that HDFS would define, and before object storage (S3) would eventually displace it.
5 Trends That Are Changing How We Do Big Data
GigaOM's 2012 survey of five trends reshaping big data: real-time processing rising against batch, NoSQL maturity, cloud-based data platforms, open-source ecosystem growth, and the shift from data collection to data monetization. A useful snapshot of where the field was heading at Hadoop's peak.
Facebook Seeks Next-Generation Big Data Tools
InformationWeek's 2012 report on Facebook's efforts to move beyond Hadoop for its next-generation big data infrastructure — one of the first major signals that the tech industry's heaviest Hadoop user was already looking past it.
New Tools Simplify the Power of Hadoop
InformationWeek's 2012 survey of tools making Hadoop more approachable for enterprise practitioners — the abstraction layers (Hive, Pig, HBase) and commercial distributions (Cloudera, Hortonworks) that were papering over Hadoop's low-level complexity. Captures the ecosystem's maturation moment.
SQL vs. NoSQL
Linux Journal's SQL vs. NoSQL comparison from the height of the NoSQL hype cycle — a useful grounding piece that distinguished the different NoSQL categories (document, key-value, column-family, graph) and when each made sense. The pendulum has since swung back toward SQL for most use cases.
How Is Big Data Faring in the Enterprise?
A ZDNet survey of how big data technologies were actually landing in enterprise environments in mid-2012 — a useful reality check during peak Hadoop hype. Most large organizations were experimenting but few had production big data systems delivering measurable business value.
Why the Days Are Numbered for Hadoop As We Know It
A 2012 GigaOM piece arguing that Hadoop's architecture had fundamental limitations that would force it to evolve or be displaced — written at the peak of Hadoop hype. Prescient in identifying YARN and the multi-framework future, but underestimated how long it would take for cloud-native alternatives to win.
How Prismatic Deals with Data Storage and Aggregation
Prismatic's 2012 engineering post on their data architecture for real-time news personalization — using Storm for stream processing, Cassandra for distributed storage, and a Clojure-based interest graph. An early look at the stream-processing stack that became standard infrastructure.