Subject
6 entries
Etl
Bookmarks
Luabase: Web3 Data Stack
Luabase is a blockchain data platform that ETLs on-chain data into queryable SQL tables — 'we ETL blockchains so you don't have to.' An early attempt to give analysts standard SQL access to blockchain activity without dealing with raw node data.
ethereum-etl: Python ETL for Ethereum Blockchain Data
ethereum-etl is the standard Python library for extracting Ethereum blockchain data into structured formats — blocks, transactions, ERC-20 transfers, receipts, logs, internal transactions — with Google BigQuery export support. The reference implementation for blockchain data pipelines.
Apollo: Cross-Chain ETL for EVM Chaindata
Apollo is a cross-chain ETL tool from Chainbound for extracting raw EVM chaindata — transactions, logs, traces — across multiple networks simultaneously. Designed for analysts who need low-level on-chain data without running their own full archive nodes.
OpenRefine
OpenRefine is an open-source desktop tool for cleaning and transforming messy data — originally Google Refine, now community-maintained. Handles clustering, deduplication, format normalization, and reconciliation against external databases like Wikidata.
Building Analytics at 500px
A first-person account of building 500px's analytics infrastructure from scratch — Amazon Redshift data warehouse, Luigi ETL, Periscope BI. The 20% evangelism rule and 'don't bake your own ETL' lesson make it one of the most practical early data engineering retrospectives.
Hadoop's Unsung Sweet Spot: Unstructured ETL
A LinkedIn discussion arguing that Hadoop's real sweet spot wasn't analytics but unstructured ETL — transforming messy, heterogeneous data into structured forms before loading into traditional data warehouses. A nuanced counterpoint to the 'Hadoop replaces SQL' narrative of 2013.