A dedicated worker for scraping NFL player, team, and game statistics from ESPN.

Sub.py watches a redis queue for tasks. Default listening is blocking. All data is stored in a PostgreSQL database.

'CRAWL ALL SPIDERS'

• Crawls all players, teams, and game spiders:
  • 'nfl_team_info'
  • 'nfl_players'
  • 'nfl_qb_stats'
  • 'nfl_rb_stats'
  • 'nfl_wr_stats'
  • 'nfl_te_stats'

'CRAWL ALL GAMES'

• Crawls all game spiders:
  • 'nfl_qb_stats'
  • 'nfl_rb_stats'
  • 'nfl_wr_stats'
  • 'nfl_te_stats'

class TeamSpider(Spider):
    name = 'nfl_team_info'

• Crawls all 2015 NFL teams. Pre-existing team data will not be overwritten.

class PlayerSpider(Spider):
    name = 'nfl_players'

• Crawls all 2015 NFL quarterbacks, running backs, wide recievers, and tight ends on active rosters. Pre-existing player information will not be overwritten.

class QbSpider(Spider):
    name = 'nfl_qb_stats'

• Crawls all 2015 quarterback regular season games and season totals. Season totals will be automatically overwritten, individual games will not be.

class RbSpider(Spider):
    name = 'nfl_rb_stats'

• Crawls all 2015 running back regular season games and season totals. Season totals will be automatically overwritten, individual games will not be.

class WrSpider(Spider):
    name = 'nfl_wr_stats'

• Crawls all 2015 wide receiver regular season games and season totals. Season totals will be automatically overwritten, individual games will not be.

class TeSpider(Spider):
    name = 'nfl_te_stats'

• Crawls all 2015 quarterback regular season games and season totals. Season totals will be automatically overwritten, individual games will not be.