-
Notifications
You must be signed in to change notification settings - Fork 0
wwoods/lamegame_tasking
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
lamegame_tasking Packages provided: lgTask Dependencies: pymongo Summary: lamegame_tasking provides a self-organizing task processing framework. The central database is a mongoDb instance. Note that all tasks run in threads in a single process. Example usage: Programming a task (must be in a file in tasks directory): class MyAppTask(lgTask.Task): def run(taskDb, arg, **kwargs): # Pass mongodb:// conn string to get a pymongo collection, database, # or connection (based on the URL). This is parameter magic that can # be overridden taskDb['collection'].insert({ 'arg': arg }) self.taskConnection.createTask( "MyAppTask" , taskDb=taskDb , arg=arg + 1 ) Programming a task that should abort if an instance is already running: class MyAppSingleton(lgTask.SingletonTask): def run(**kwargs): # ...Normal run function The task is singleton'd based on the "taskName" parameter, which for classes defaults to the class name. If you have e.g. a per-database singleton, simply make scheduling or creating the task extract the database name and initialize "taskName" accordingly. === Event-Drive Tasks === Creating a sample task to run: >> import lgTask >> c = lgTask.Connection() >> c.createTask("MyAppTask"[, taskName="MyAppTaskAltName"], arg="Hello, world!") Creating a task to run at some point in the future: >> import lgTask >> c = lgTask.Connection() >> c.delayedTask(datetime object OR "1 day/hour/minute/second" , "MyAppTask", arg="Delayed!") === Scheduled and batched tasks === Creating a batched task (if already scheduled, don't schedule again; otherwise schedule the task): >> import lgTask >> c = lgTask.Connection() >> c.batchTask(datetime object OR "1 day/hour/minute/second", "MyAppTask") If the task was already scheduled (and not executed yet), these functions do nothing; that task is presumed to cover whatever this newly requested task is supposed to accomplish. Be careful with that. But this is mostly for batching things when batches are gradually built in the db. Creating a recurring scheduled task (executed immediately and at given intervals): >> import lgTask >> c = lgTask.Connection() >> c.intervalTask(datetime object OR "1 day/hour/minute/second", "MyAppTask") If an intervalTask is called with a name that already exists but different kwargs or interval, this will raise an exception of type TaskKwargError rather than rewriting that task's schedule / kwargs. If called with identical arguments to the existing, does nothing. Unscheduling a scheduled task (does not raise exception if the task does not exist): >> c.intervalTaskRemove("MyAppTask"[, taskName="name"]) === Task Processors === Configuring and running processors for development: At the beginning of your app's web server / task generator: >> import lgTask >> processor = lgTask.Processor("/path/to/config"[, taskName="myProcessor"]) >> processor.start() This will spawn the processor in a new thread. If you want to specify the config in code, you can pass a dict or lgTask.lib.reprconf.Config object instead of a file path. If you want to block on the processor until it can no longer consume any tasks, use: >> processor.stop(onNoTasksToConsume=True) Configuring and running processors for production: 1. Symlink lamegame_tasking/bin/lgTaskProcessor to your PATH (or execute directly) 2. Setup a user to run the task processors; in their home directory, check out your task source code and dependencies. 2. Setup a configuration file for your deployment, like the following (myAppProcessor.cfg): [processor] # This name is required; on the bright side, this allows you # to put other configuration items in the file taskDatabase = 'mongodb://[user:password]@server:port[/mytaskdb]' pythonPath = [ '../' ] # Folders relative to config file to add to python path # when running the Processor. 3. Setup cron to run lgTaskProcessor every minute (to prevent it from erroring out; a processor will automatically not run two instances) with the following parameters: a. Config file - absolute path to task processor configuration b. [optional] Number of processors to spawn - if unspecified, spawn 1. z. Example: lgTaskProcessor /home/user/myAppProcessor.cfg 2 Each Processor runs one task at a time. They are named according to the format "hostname-index", with index starting at 1. If only one processor is spawned b y the lgTaskProcessor dscript, the "-index" portion of the name is dropped.
About
A self-organizing, distributed task processing framework on top of mongodb and python
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published