dev@ubuntu:~$ pip install structure_spider>=0.9.10
dev@ubuntu:~$ startproject myapp
New structure-spider project 'myapp', using template directory '/home/dev/.pyenv/versions/3.6.0/lib/python3.6/site-packages/structor/templates/project', created in:
/home/dev/myapp
You can start the spider with:
cd myapp
custom-redis-server -ll INFO -lf
scrapy crawl douban
dev@ubuntu:~$ custom-redis-server -ll INFO -lf
dev@ubuntu:~$ cd myapp/myapp/
dev@ubuntu:~/myapp/myapp$ ls
items settings.py spiders
dev@ubuntu:~/myapp/myapp$ createspider -s taobao id title brand price colors images
TaobaoSpdier and TaobaoItem have been created.
dev@ubuntu:~/myapp/myapp$
参考资料:使用structure_spider多请求组合抓取结构化数据
dev@ubuntu:~/myapp/myapp$ scrapy crawl taobao
dev@ubuntu:~/myapp$ feed -s taobao -c test -uf myapp/text.txt --custom # --custom代表使用的是简单redis
更多资源: