File:mass-email-delivery-code/README
Author: Naomi Fox <naomi.fox@gmail.com>
Date: Dec 8, 2011, Januar 22, 2011

= Description =
Set of files for submitting form-based emails to members of congress.

To get a copy of this code:
 git clone git://github.com/naomifox/mass-email-delivery-code.git

= Dependencies =
== Webpy ==
 
 git clone git://github.com/webpy/webpy.git
 cd mass-email-delivery-code 
 ln -s ../webpy/web .

== BeautifulSoup ==
 sudo easy_install BeautifulSoup 

== ClientForm ==
 sudo easy_install ClientForm

= Testing =

To run unittests:
 python TestWriteYourRep.py

To run on one senator:
  python WriteYourRep.py stest senator - example: stest boxer

To run for one district:
 python WriteYourRep.py htest dist - example: stest MA-01

To run test over all senators:
 python main.py stest

To run test over all reps:
 python main.py htest

= Delivering emails for a large number of signers in a form =

(1) Download data as a csv from the blue state digital site or Action Kit.  The first line should be a header with column names.

(2) Create an Amazon Web Services account: http://aws.amazon.com/

(3) Set up an EC2 spot instance (be sure to choose one with more memory, and not the micro).  
With the 64 big Amazon Linux m1.xlarge (8 ECUs, 4 Cores, 15 G memory), we are able to send > 10,000 emails per hour.

(4) Launch the EC2 instance, scp data over and install the code and dependencies (follow installation instructions above).
For Amazon Linux, you must install git first: sudo yum install git

(5) Connect to your EC2 instance via ssh.  Split up your csv file into smaller chunks using the split script.  

For example, to split "my-data.csv" into csv files with headers and 5000 lines each:
    python utils/splitCsv.py my-data.csv 5000

(6) Start multiple jobs.
    	  rm runAll.sh
    	  for f in x??; do echo "nohup python EmailHouseOrSenateFromCSV.py house $f messagefile $f-house.stat &> $f-house.stdout &" >> runAll.sh; done
    	  for f in x??; do echo "nohup python EmailHouseOrSenateFromCSV.py senate $f messagefile $f-senate.stat &> $f-senate.stdout &" >> runAll.sh; done
    Start a screen session, so you can log out and the jobs will continue to run.	
    	  /bin/sh runAll.sh

You can monitor how many emails have been sent out by doing a line count on your status file.
    wc -l data/*stat

= Updating =

When a member changes, or the url to the contact page changes, local data files need to be updates.

The utils/ directory contains a script, build.sh, to update all databases.

Test the contact links by running:
python main.py stest

Then view the file senate_test_out.txt for failures.  Attempt to find the correct contact links for those that failed and update senators_cfm.xml

Test contact links and update as needed:
python main.py htest

= To do =

1. Write code to split up signer data by state, and potentially by district.  
In this way, we can control the rate that the reps' servers are receiving emails, and might help with the HTTP 403 errors.

2. Write whole row for failures into a file, to make it easier to resend emails again.