Skip to content

Improved deferred library implementation for Google App Engine

License

Notifications You must be signed in to change notification settings

modrzew/appengine-improved-deferred

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Improved deferred for App Engine

Build Status Coverage Status

This is homemade implementation of deferred.defer for Google App Engine, specifically for Python runtime.

Why?

Original defer, while being extremely useful in the early stage of development (you just defer the function and forget about it), is not very friendly to your app when it comes to monitoring resources usage, or finding any particular error in the logs. This is how your logs screen will look like after a while:

    [200] /_ah/queue/deferred
    [400] /_ah/queue/deferred
    [200] /_ah/queue/deferred
    [200] /_ah/queue/deferred
    [200] /_ah/queue/deferred
    [500] /_ah/queue/deferred
    [200] /_ah/queue/deferred
    [200] /_ah/queue/deferred
    [503] /_ah/queue/deferred
    [200] /_ah/queue/deferred
    ...

And in the time of writing this code, Cloud Trace was in beta phase. It's an tool for App Engine for creating reports about resource usage on any particular handler in your app. It's not very helpful if all of your handlers are identified by /_ah/queue/deferred, is it?

Improved deferred turns your logs into something like this:

    [200] /deferred/user/create
    [400] /deferred/item/remove
    [200] /deferred/cron_job
    [200] /deferred/user/index
    [200] /deferred/item/add
    [500] /deferred/item/add
    [200] /deferred/user/index
    [200] /deferred/user/create
    [503] /deferred/cron_job
    [200] /deferred/relationship/create
    ...

Now it's tremendously easy to find the request you had in your mind!

Usage

Instead of importing google.appengine.ext.deferred, just import deferred module from this repository!

import deferred


def some_function():
    pass


# Use it as you normally would use defer
deferred.defer(some_function, 1, 2, some='kwarg', _target='module')


# And just decorate your functions with @deferred for magic!
@deferred.deferred('functions/magic')
def magic_function():
    pass


# It's simple!
deferred.defer(magic_function, 3, 4, another='kwarg', _queue='not-default')

Where is the magic?

@deferred decorator defers your functions using taskqueue.add and special handler instead of just calling deferred.defer. That gives you much more control over what is executed, how and when. And how it's displayed in the logs or in the reports.

Another thing worth mentioning is that Datastore is never used here. Standard deferred library pickles your task payload, and, if it's bigger than payload limit for the taskqueue (100KB at the moment of writing this) it puts it to Datastore. Which slows your request down. By using @deferred, you need to monitor what you pass to the deferred tasks (rule of thumb is, you should pass urlsafes/IDs instead of actual objects).

License

See LICENSE.md.

About

Improved deferred library implementation for Google App Engine

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages