Scrapy

scrapy-history-middleware: S3 Historical Cache for Scrapy

A Scrapy middleware that stores every crawled response in S3, building a historical archive of web resources over time. Enables point-in-time replay of crawls and separates the concerns of fetching from processing.