Tools to get tweets in a city and save them to PostgreSQL.

s3_readme.md: Info about how to find the data we collected from this project. It's stored as gzipped json files in a public bucket on s3.

scrape_tweets_in_area.py: the main thing. Run this with --city=(city name) and it'll listen to the Twitter streaming endpoint for a small bounding box around that city (defined beforehand).

scrape_instagram_in_area.py: the same, but for instagram. But limited to Pittsburgh, so far. EDIT: as of September 2016, this won't work; they shut down the bit of the API we were using to grab instagram photos.

notify_if_broken.py: sends an email if no data has been added to each table in the database in 24 hours. (run via cron.)

print_table_counts.py: prints out how many items are in each table. (run via cron to make a log of sorts.)

create_tables.py: if you're starting on a new computer, this can help you set up the database. Some basic instructions included in the top comments in this file too.

utils.py: common functions used in more than one of the above files.

crontab: my crontab on Domo (our EC2 machine) that runs a couple of the above daily.