django-cachetree
provides caching of configurable trees of related model instances in Django. For example, with django-cachetree
you could easily cache a user instance and the user's photos and comments on a photo-sharing site as a single item in the cache. When a user is fetched from the database, its related objects are "prefetched" before the user object is set in the cache. This means that when you retrieve the user object from the cache, the related objects are there with it and you can access them without hitting the database (or hitting the cache again):
user = User.objects.get_cached(pk=1) # user not yet in cache, hits the
# database and prefetches related objects
....
user = User.objects.get_cached(pk=1) # hits the cache
photos = user.photo_set.all()
print photos[0].title # doesn't hit database or cache
comments = user.comment_set.all()
print comments[0].date # doesn't hit database or cache
You can configure django-cachetree
to cache related objects of related objects of related objects, to any depth. For example, on a blog, you could cache author objects along with their set of blog entries, the set of comments for each of those entries, and the commenter for each of those comments.
django-cachetree
automatically invalidates cached objects when they or the related objects cached with them are changed, deleted, or created.
django-cachetree
requires Django 1.3 and Python 2.5, 2.6, or 2.7.
You can install django-cachetree
with :
pip install django-cachetree
or :
easy_install django-cachetree
This will add cachetree
to your Python path. Add 'cachetree'
to your INSTALLED_APPS
, and add 'cachetree.middleware.InstallCachetree'
to your middleware classes. This will add a get_cached
method to the default manager for each of your cached models. (Note that because get_cached
is added to the manager class, it will be available on all instances of that manager class. However, attempting to use it on a model not defined in your CACHETREE
setting will raise a ValueError
.) If invalidation is enabled, cachetree.install()
also registers signal handlers that are used for invalidation.
Installing cachetree
via middleware will only work if there is a page request to trigger the middleware. To install and use cachetree
outside of a request-response context (for example, in a test suite), run :
cachetree.install()
in your testrunner or at the bottom of your models file.
To use django-cachetree
, add a CACHETREE
setting to settings.py
. The CACHETREE
setting consists of nested dictionaries that tell django-cachetree
what to cache. The keys in the topmost dictionary should contain the app_label
for each app that has models you wish to cache. Each app_label
's dictionary should contain a key with the class name (as a string) for each cached model in the app. These are the models whose managers will provide a get_cached
method, and are known as "root models" in django-cachetree
's terminology. For example, to cache Author
and Entry
models but none of their related objects in an app called myapp
, you would write:
CACHETREE = {
"myapp": {
"Author": {},
"Entry": {},
}
}
The dictionary for each root model can contain three optional keys, "timeout"
, "lookups"
, and "prefetch"
.
timeout
The timeout, in seconds, to use when caching instances of this model. Overrides your global timeout setting in
CACHES
.lookups
A tuple containing the field names that can be used as kwargs when calling
get_cached
for this model. By default, lookups are allowed by primary key. If your model's primary key field isid
, the default setting would be("pk", "id")
.To lookup by a combination of fields, include the field names as a tuple within your
lookups
tuple. For example, to look upUser
instances byid
or byfirst_name
andlast_name
:CACHETREE = { "auth": { "User": { "lookups": ( "id", ("first_name", "last_name"), ) } } }
If invalidation is enabled, lookups are restricted to fields defined on the model, including
ForeignKey
fields andOneToOneField
s but excludingManyToManyField
s. SpecifyingManyToManyField
s or reverseForeignKey
orOneToOneField
s will raisecachetree.ImproperlyConfigured
. Lookup separators (for example,username__contains
) are also not allowed and will raiseImproperlyConfigured
. To know what keys to invalidate,django-cachetree
requires exact lookups (which is the default when no lookup separator is used).prefetch
A dictionary specifying the tree of related objects to prefetch and cache with the root model instance. Each key should be the attribute name (as a string) of the related instance(s) to be prefetched. Each key's value should be a dictionary of attribute names to prefetch on the related instance(s), or an empty dictionary (or
None
) if no further relationships should be prefetched. Any relationship can be prefetched:OneToOneField
,ForeignKey
, andManyToManyField
, forward or reverse. For example, to cache author objects, their set of entries, those entries' comments, and each comment's commenter, you might write:CACHETREE = { "myapp": { "Author": { "lookups": ( "pk", "id", ("first_name", "last_name"), ), "prefetch": { "entry_set": { "comment_set": { "commenter": {}, }, }, }, }, }, }
The above example assumes that each
Author
object is related to its entries by anentry_set
attribute, each entry object is related to its comments by acomment_set
attribute, and each comment object relates to its commenter by acommenter
field.If invalidation is enabled, there is one restriction on prefetching. If you prefetch a
ManyToManyField
(forward or reverse) that defines a custom intermediary model (as specified with thethrough
argument on the model field definition), you must also prefetch the attribute that points to the intermediary instances. For example, if you have anEntry
model related to aCategory
model through a custom intermediary model calledEntryCategory
, and you prefetchEntry.categories
(aManyToManyField
), you must also prefetch theEntry.entrycategory_set
attribute that Django adds to yourEntry
model, orImproperlyConfigured
will be raised.
You can find example CACHETREE
settings in django-cachetree
's test module, which defines models and settings covering all possible relationships.
When you configure django-cachetree
to cache a ManyToManyField
or reverse ForeignKey
, such as user.photo_set
(where Photo
has a foreign key to User
), django-cachetree
calls user.photo_set.all()
, evaluates the queryset, and caches the results on the user
when prefetching. Subsequent calls to user.photo_set.all()
will return the cached results, rather than returning a new queryset (which would require hitting the database again to evaluate). django-cachetree
patches the manager on ManyToManyField
and ForeignKey
descriptors to make this behavior possible. However, only the all()
method is patched. If you call user.photo_set.count()
or user.photo_set.filter()
or any other method besides all()
, you will bypass the cached results and hit the database. Assuming your object set is not huge, you can avoid hitting the database by calling all()
and counting or filtering the results within your code.
When you call cachetree.install()
, django-cachetree
analyzes your CACHETREE
setting and determines which relationships must be followed in order to traverse the tree backwards from prefetched related instances to their root model instances. Using this information, whenever a model defined in your CACHETREE
setting (either as a root model or as a prefetched relationship) is created, saved, or deleted (and in the case of ManyToManyField
s and reverse ForeignKey
s, added, removed, or cleared using the field manager's add()
, remove()
, or clear()
methods), django-cachetree
traverses its relationships back to the root model instance(s) that need to be invalidated. django-cachetree
uses a post_init
signal handler to keep track of each instance's initial state, and when the instance changes and is saved, django-cachetree
follows both the instance's new and initial values to find the root model instances that need to be invalidated. For example, if you cache Author
objects along with their entry_set
, and you change an Entry
object's author, django-cachetree
will invalidate both the new and the initial Author
objects for that entry.
Important Caveat: django-cachetree
does not perform invalidation when you run an UPDATE
query using a manager's update()
method. You will either need to invalidate the affected instances yourself by calling invalidate()
(described below), rely on the cached objects to expire naturally, or avoid using update()
.
If django.contrib.auth
is installed in your project, you can use django-cachetree
's authentication backend:
AUTHENTICATION_BACKENDS = (
"cachetree.auth.CachedModelBackend",
)
This will look in the cache before hitting the database when authenticating users. Adding the auth.User
model to your CACHETREE
setting is optional. Not adding it implies the following settings:
CACHETREE = {
...
"auth": {
"User": {
"lookups":(
"pk",
"username",
),
},
},
}
If you wish to allow additional lookups on User
or to prefetch related instances, explicitly define User
in your CACHETREE
setting.
The following functions can be imported from cachetree
:
get_cached_object_or_404
Works like
get_object_or_404
, but usesget_cached
instead ofget
.invalidate(*instances)
Traverses relationships on each of the
instances
to find and invalidate its root model instance(s).no_invalidation
Decorator that disables invalidation for the duration of the function it decorates.
CACHETREE_DISABLE
Set to
True
to disabledjango-cachetree
. Calls toget_cached()
orget_cached_object_or_404()
will useget()
. Calls toinvalidate()
and uses of theno_invalidation
decorator will have no effect. This allows you to temporarily disabledjango-cachetree
without modifying any code. Default:False
.CACHETREE_INVALIDATE
Set to
False
to disable invalidation.django-cachetree
will continue to cache model objects but will not invalidate them when they change. Calls toinvalidate()
and uses of theno_invalidation
decorator will have no effect. Default:True
.CACHETREE_MANY_RELATED_PREFIX
Controls the prefix
django-cachetree
uses when it prefetches a set of related objects and caches it on a model instance. In the example ofauthor.entry_set.all()
,django-cachetree
caches the author's set of entries asauthor._cached_entry_set
, and subsequent calls toauthor.entry_set.all()
return this attribute. Normally you will not need to access this attribute directly, but this setting allows you to change the prefix in case of name conflicts. Default:_cached_
.