Arachnado is a tool to crawl a specific website. It provides a Tornado-based HTTP API and a web UI for a Scrapy-based crawler.

License is MIT.

Python 2.7 is required to run server. To build static assets node.js + npm are required.

Install all Python requirements from requirements.txt using pip:

pip install -U -r requirements.txt

Install all JavaScript requirements using npm. Run the following command from the repo root:

npm install

then rebuild static files (we use Webpack):

npm run build

or auto-build static files on each change during development:

npm run watch

To run Arachnado execute the following command from the repo root:

python -m arachnado

Then visit http://127.0.0.1:8888/