Skip to content
This repository has been archived by the owner on Mar 18, 2024. It is now read-only.

hack4code/spider

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Blog Spider

Run

  • docker-compose up --build -d

Cron Job

run spiders

  • curl -X POST -H "Content-Type: application/json" -d '{"spiders": ["all"]}' http://127.0.0.1:8080/submit/crawl

backup mongo

  • docker-compose exec mongodb mongodump -u scrapy -p scrapy -d scrapy -c spider --gzip --archive=/dump/spider.gz
  • docker-compose exec mongodb mongodump -u scrapy -p scrapy -d scrapy -c category --gzip --archive=/dump/category.gz

React JS Build

  • docker container run --rm -v `pwd`/www/src/static/:/static/ -w /static/app/ node ./build.sh

About

spiders crawl blogs (rss | atom | blog)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published