The following setup will scrape all of the current issue pages for AJM's various journals to search for new articles. If new articles are found then a summary email is created and sent to anyone who is on the recipients list.

With some modification I'm sure it could be adapted to scrape any other page and automatically email a list of recipients.

All programmed in Python

Scrapy > MongoDB > Jinja2 > Email delivery with SMTPLib via gmail

• MongoDB must be installed and mongod running.
• $ pip install -r requirements.txt
• The script depends on a file that I've excluded from git which has the gmail login details and a list of the recipients. It is located in the root directory of the project and is of the form:

  recipients = [
      {
          'name': 'Oliver',
          'email': 'dummy-email1@gmail.com'
      },
      {
          'name': 'Nollie',
          'email': 'dummy-email2@gmail.com'
      }
  ]
  
  GMAIL_CREDENTIALS = {
      'email': 'senders email address',
      'username': 'gmail username',
      'pass': 'gmail password'
  }

• Run $ python main.py
• Setup a cron job to run the script at any interval that you desire.