Your friendly neighborhood S3 helper. It knows just a few tricks, but hopefully they'll help you out.

• Automatic retries -- with exponential backoff to make your life easier
• Parallel batch operations -- using gevent to make it hella fast
• Built-in S3 Mocking -- to make your testing eaiser
• Automatic multipart uploading -- and it can retry each piece individually

This module requires that you have boto installed. After that:

sudo python setup.py install

s3po knows a few tricks, but at its core, you'll use two methods: upload and download:

import s3po
conn = s3po.Connection()

# Upload with a string
conn.upload('bucket', 'key', 'howdy')
# Upload a file
with open('foo.txt') as fin:
    conn.upload('bucket', 'key', fin)

# Download as a string
print conn.download('bucket', 'key')
# Or into a file
with open('foo.txt', 'w') as fout:
    conn.download('bucket', 'key', fout)

The batch object works like a context manager that provide the same interface as the connection object. The only difference is that all the requests run in a gevent pool. When the context is closed, it waits for all functions to finish.

# Upload these in a gevent pool
keys = ['key.%i' for key in range(1000)]
with conn.batch(50) as batch:
    for key in keys:
        batch.upload('bucket', key, 'hello')

# And now they're all uploaded

When you want to take some action with the result, all the functions have an additional callback optional parameter to you can grab the result. This is particularly useful when doing bulk downloads:

from functools import partial

def func(key, value):
    print 'Downloaded %s from %s' % (key, value)

with conn.batch(50) as batch:
    for key in keys:
        batch.download('bucket', key, callback=partial(func, key))

If the provided data is sufficiently large, it will automatically run the upload as a multipart upload rather than a single upload. The advantage here is that for any part upload that fails, it will retry just that part.

You can turn on mocking to get the same functionality of s3po that you'd expect but without ever having to touch S3. Use it as a context manager:

# This doesn't touch S3 at all
with conn.mock():
    conn.conn.create_bucket('foo')
    conn.upload('foo', 'bar', 'hello')

If you're writing tests, this is a common pattern:

class MyTest(unittest.TestCase):
    def setUp(self):
        self.s3po = Connectino()
        self.mock = self.s3po.mock()
        self.mock.start()
        self.s3po.conn.create_bucket('bucket')

    def tearDown(self):
        self.mock.stop()