‹
   €     |    |     €€€€≤±‹€€€€≤±‹
   €    .| -- | -+
   €   ' |    |  |  €€€€≤±±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   €    `|    |
   €     |`.  |     €€€€≤±±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   €     |  `.|
   €     |    |`.   €€€€≤±±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   € +   |    |   '
   € | _ | __ | _'   €€€≤±±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   €     |    |
   €€€€€€€€€€€€€€€€€€€€€€≤±±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   €
   €€€€€€€€€€€€€€€€€€€€€€≤±±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   €
   €                      fl±≤€€€€€€€€€€€€≤±±≤€€€€€≤±
   €                                      fl±≤€€€€€≤±
   €
   €

federal-treasury-api is the first-ever electronically-searchable database of the Federal government's daily cash spending and borrowing. It updates daily and the data can be exported in various formats and loaded into various systems.

You can run this on any number of servers, but we happen to be using ScraperWiki. You can check out their documentation here

To use ScraperWiki, log in here, make a project, click the "SSH in" link, add your SSH key and SSH in. Then you can SSH to the box like so.

ssh cc7znvq@premium.scraperwiki.com

Or add this to your ~/.ssh/config

Host fms
HostName premium.scraperwiki.com
User cc7znvq

and just run

ssh fms

Some notes about how ScraperWiki works:

• We have a user account in a chroot jail.
• We don't have root, so we install Python packages in a virtualenv.
• Files in /home/http get served on the web.
• The database /home/scraperwiki.sqlite gets served from the SQLite web API.
  • NOTE: the home/scraperwiki.sqlite is simply a symbolic link to data/treasury_data.db generated by this command: ln -s data/treasury_data.db scraperwiki.sqlite

The directions below still apply for any other service, of course.

Optionally set up a virtualenv. (You need this on ScraperWiki.) Run this from the root of the current repository.

virtualenv env

Install dependencies.

pip install -r requirements.pip

Enable the git post-merge hook.

cd .git/hooks
ln -s ../../utils/post-merge .

This one command downloads the (new) fixies and converts them to an SQLite3 database.

./run.sh

Run everything

cd parser
python download_and_parse_fms_fixies.py

Run crontab -e to edit the schedule later.

To unit test parser functions, write tests in tests/parse_fms_fixies.py, and then run them with nose.

cd tests
nosetests

You might want to write some tests to make sure that a specific table is perfectly parsed. In order to do that, copy the source fixie and the intended csv to the fixtures directory. (Name them as they would be named in the data folder.) A test will be generated for each of these.

To make sure that the script is still running on ScraperWiki, run

python tests/is_it_running.py

To make sure that results are reasonable, run the integrity tests.

cd tests
python integrity_tests.py

Results will appear in tests/test_output/%Y-%m-%d.csv, named after today's date.

If you prefix a command with ./monitor.sh and the command fails, the output of the command will be sent wherever you specify. In order to use this, you need to create a ~/.smtpcli.conf.

There are eight tables.

• I. Operating Cash Balance (t1)
• II. Deposits and Withdrawals (t2)
• IIIa. Public Debt Transactions (t3a)
• IIIb. Adjustment of Public Dept Transactions to Cash Basis (t3b)
• IIIc. Debt Subject to Limit (t3c)
• IV. Federal Tax Deposits (t4)
• V. Short-term Cash Investments (t5)
• VI. Incom Tax Refunds Issued (t6)

examples contains some example downloader functions. csv/json is a website that serves the data as csv.

If (and only if) you change the parsing engine, you can reset the data by running reset_data.sh. This function will attempt to delete data/daily_csv, data/lifetime_csv, data/treasury_data.db, and then run python parser/download_and_parse_fms_fixies.py

If you have a strange error, try deleting the parser output and updating the code to the latest master. You can do this by running ./reset_data.sh.