Collecting StatisticsΒΆ

You can store different scraping statistics for the spiders into the PostgreSQL database. To enable it set:

PIPELINE_USE_ITEMCOUNT_POSTGRES=1
STATS_CLASS=skyscraper.statscollectors.PostgresStatsCollector

Warning

These option names are probably not stable, yet. We might combine them into one option and rename the option name to something like SKYSCRAPER_STATS_POSTGRES.