scrapy-dotpersistence 0.3.0

安装

您可以使用pip安装scrapy-dotpersistence

pip install scrapy-dotpersistence

然后您可以在您的settings.py中启用此扩展

EXTENSIONS = {
    ...
    'scrapy_dotpersistence.DotScrapyPersistence': 0
}

如何使用它

通过settings.py启用扩展

DOTSCRAPY_ENABLED = True

通过settings.py配置扩展

ADDONS_AWS_ACCESS_KEY_ID = "ABC"
ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"
ADDONS_AWS_USERNAME = "username"
ADDONS_S3_BUCKET = "test-bucket-name"

您可以使用环境变量更改dotpersistence文件夹路径

export DOTSCRAPY_DIR='/tmp/.scrapy'