Skip to content

invertpyramid/s-whoscored

Repository files navigation

A Spider for WhoScored

This is a scrapy project for WhoScored web scraping.

Overview

pylint Score

image

License: AGPL v3

Code style: black

Requirements

Python 3

pyup

Known Vulnerabilities

Renovate enabled

  • Python 3.6+
  • Scrapy 1.6.0
  • Fully tested on Linux, but it should works on Windows, Mac OSX, BSD

Usage

Run Sentry

Initial postgres with senty first:

1. Generate secret key first: :: docker run --rm sentry config generate-secret-key

2. Use the secret key to create a database in postgres: :: docker run --detach --name sentry-redis-init --volume $PWD/redis-data:/data redis docker run --detach --name sentry-postgres-init --env POSTGRES_PASSWORD=secret --env POSTGRES_USER=sentry --volume $PWD/postgres-data:/var/lib/postgresql/data postgres docker run --interactive --tty --rm --env SENTRY_SECRET_KEY='<secret-key>' --link sentry-postgres-init:postgres --link sentry-redis-init:redis sentry upgrade

Then input the superusername and password

3. Stop the redis and postgres: :: docker stop sentry-postgres-init sentry-redis-init && docker rm sentry-postgres-init senty-redis-init

  1. Edit the env files to add the superusername, password and database related information

5. Start sentry with docker-compose.yml: :: docker-compose up --detach && docker-compose logs --follow

Run MongoDB for httpcache

Run Percona Server MongoDB for cookies

Debug with mitmproxy

Start mitmproxy

Enable httpproxy middleware

Debug in mitmproxy

Server-side Replay ~~~~~~~~~~~~~~~~~

Follow Features of mitmproxy:

https://docs.mitmproxy.org/stable/overview-features/#server-side-replay

TODO