MaterialGirl is a library to keep materialized views and consolidations up-to-date.
Intalling it is as easy as::
$ pip install materialgirl
MaterialGirl can be used to keep a set of data, called materials, up-to-date.
It is very useful to keep slow queries or consolidated data fast to get.
You also need to select a storage (more on storages later). MaterialGirl comes bundled with two storages:
- materialgirl.storage.memory.InMemoryStorage
- materialgirl.storage.redis.RedisStorage
Using MaterialGirl:
from materialgirl import Materializer
from materialgirl.storage.memory import InMemoryStorage
def get_very_slow_data():
return 'this is very slow to get'
storage = InMemoryStorage()
girl = Materializer(storage=storage)
girl.add_material(
'my-very-slow-data-key',
get_very_slow_data, # this should be the function to get up-to-date data
120 # the expiration in seconds
)
girl.run() # this updates all the expired materials and should be run in a loopTo run the loop you can use whatever daemon, worker, runner solution you'd like. We recommend Sheep.
Whenever you need to get the up-to-date information you set with material girl, just call the get method on an instance with the same materials as the one you are running to update the data, like this:
from materialgirl import Materializer
from materialgirl.storage.memory import InMemoryStorage
def get_very_slow_data():
return 'this is very slow to get'
storage = InMemoryStorage()
girl = Materializer(storage=storage)
girl.add_material(
'my-very-slow-data-key',
get_very_slow_data, # this should be the function to get up-to-date data
120 # the expiration in seconds
)
assert girl.get('my-very-slow-data-key') == 'this is very slow to get'MaterialGirl is lazy. If it has not the up-to-date value in storage to give you, it will call your get method, update the storage and return the proper value.
Sometimes you may need an information just before material girl updates it. In such cases you may have a cache miss. To avoid this, you can define a grace period of time for the cached information:
import time
from materialgirl import Materializer
from materialgirl.storage.memory import InMemoryStorage
def get_very_slow_data():
return 'this is very slow to get'
storage = InMemoryStorage()
girl = Materializer(storage=storage)
girl.add_material(
'my-very-slow-data-key',
get_very_slow_data, # this should be the function to get up-to-date data
120, # the expiration in seconds
240 # the grace period in seconds
)
time.sleep(140)
assert girl.get('my-very-slow-data-key') == 'this is very slow to get'This may not be the most up-to-date information, but cache misses become rare.
The in-memory storage is more of a proof of concept and should only be used for very simple scenarios as it stores the data in-process.
The redis storage takes one parameter: a redis connection.
import redis
from materialgirl.storage.redis import RedisStorage
storage = RedisStorage(redis=redis.StrictRedis())Creating a custom storage is as simple as implementing two methods: store and retrieve:
from materialgirl.storage import Storage
class MyCustomStorage(Storage):
def store(self, key, value, expiration=None):
# store the value somewhere under `key`
def retrieve(self, key):
# retrieve the value for `key` from same storage solutionJust fork, commit, pull request our way.
MIT licensed.