Scrapy extension to sync .scrapy folder to an S3 bucket.
You can install scrapy-dotpersistence using pip:
pip install scrapy-dotpersistenceYou can then enable the extension in your `settings.py`:
EXTENSIONS = {
...
'scrapy_dotpersistence.DotScrapyPersistence': 0
}Enable extension through `settings.py`:
DOTSCRAPY_ENABLED = TrueConfigure the exension through `settings.py`:
ADDONS_AWS_ACCESS_KEY_ID = "ABC"
ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"
ADDONS_AWS_USERNAME = "username"
ADDONS_S3_BUCKET = "test-bucket-name"You can change a dotpersistence folder path with environ:
export DOTSCRAPY_DIR='/tmp/.scrapy'