This script fetches blocklist .json from urls (such as shavar-prod-lists) and generates safebrowsing-compatible digest list files to be served by shavar.

• python 2.x
• (optional) virtualenv and/or virtualenvwrapper

1. (optional) Make a virtualenv for the project and activate it:

   virtualenv shavar-list-creation
   source shavar-list-creation/bin/activate
   

2. Install required libraries:

   pip install -r requirements.txt
   

3. Copy the sample_shavar_list_creation.ini file to shavar_list_creation.ini:

   cp sample_shavar_list_creation.ini shavar_list_creation.ini
   

4. Run the lists2safebrowsing.py script:

   ./lists2safebrowsing.py
   

This is run by a Jenkins deployment job every 30 minutes that:

1. Checks out this repository
2. Checks out the shavar-list-creation-config repository
3. Copies stage.ini or prod.ini to shavar_list_creation.ini
4. Runs python lists2safebrowsing.py, which uploads updated safebrowsing list files to S3 for shavar.