torspider

It does things that crawl Tor.

Initial ideas inspired by terrible jokes on Discord about Tor analytics. Lots of help with not reinventing the code for the crawling wheel comes from this crawler.

Licence is AGPL.

Notes

docker-compose run --rm spider python init_db.py - Init the DB
docker-compose up --scale spider=4 -d brings some nice multispider crawling
Rebloom is a required Redis module for duplicate URL filtering.
It is assumed that POSTGRES_URL is a bouncer that does its own pooling such as pgbouncer/pgpool.
Postgres MUST be the database due to Postgres specific features.

Name		Name	Last commit message	Last commit date
Latest commit History 186 Commits
.vscode		.vscode
alembic		alembic
containers		containers
spidercommon		spidercommon
tests		tests
tools		tools
torspider		torspider
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.envrc.example		.envrc.example
.gitignore		.gitignore
.gitmodules		.gitmodules
Dockerfile		Dockerfile
LICENCE.md		LICENCE.md
README.md		README.md
alembic.ini		alembic.ini
docker-compose.production.yml		docker-compose.production.yml
init_db.py		init_db.py
requirements.txt		requirements.txt
scrapy.cfg		scrapy.cfg
setup.py		setup.py

License

general-programming/torspider

Folders and files

Latest commit

History

Repository files navigation

torspider

Notes

About

Resources

License

Stars

Watchers

Forks

Languages