Skip to content

gitaaron/slow-pipeline

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 

Repository files navigation

Slow Pipeline Demo

This is a standalone scrapy project that demonstrates what happens if persisting items in the item pipeline takes longer than the rate of extraction.

To run

Execute : ./run_demo.py

To monitor bubbling items

You can use the trackref module either through the telnet console or via another means.

About

A standalone scrapy project that reproduces item bubbling due to slow processing in the pipeline.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages