def tasks():
steps = [
'intro',
'one_time', 'repeats', 'repeats_failed',
'group_names', 'uuid', 'futures', 'priority',
]
docs = Storage()
comments = Storage()
docs.intro = """
#### Intro
So, here we are trying to learn (and test) web2py's scheduler.
Actually you have to download latest trunk scheduler to make it work (backup current gluon/scheduler.py and replace with the one on trunk).
This app ships with a default SQLite database, feel free to test on your preferred db engine.
All examples code should work if you just prepend
``
import datetime
from gluon.contrib.simplejson import loads, dumps
sr = db.scheduler_run
sw = db.scheduler_worker
st = db.scheduler_task
``:python
DRY!
Additionally, every example uses ``task_name``, but that is not a required parameter.
It just helps **this app** to verify that all is working correctly when you press the **Verify** button.
We have 3 functions defined into models/scheduler.py (don't get confused). It should be something like this:
``
# coding: utf8
import time
from gluon.scheduler import Scheduler
def demo1(*args,**vars):
print 'you passed args=%s and vars=%s' % (args, vars)
return args[0]
def demo2():
1/0
def demo3():
time.sleep(15)
print 1/0
return None
def demo4():
time.sleep(15)
print "I'm printing something"
return dict(a=1, b=2)
def demo5():
time.sleep(15)
print "I'm printing something"
rtn = dict(a=1, b=2)
scheduler = Scheduler(db)
##or, alternatively :
#scheduler = Scheduler(db,
# dict(
# demo1=demo1,
# demo2=demo2,
# demo3=demo3,
# demo4=demo4,
# foo=demo5
# )
# )
``:python
So, we have:
- demo1 : standard function, with some printed output, returning the first arg
- demo2 : never returns, throws exception
- demo3 : sleeps for 15 seconds, tries to print something, throws exception
- demo4 : sleeps for 15 seconds, print something, returns a dictionary
- demo5 : sleeps for 15 seconds, print nothing, doesn't return anything
- demo6 : sleeps for 20 seconds, catches the sigterm signal
The scheduler istantiated with the db only. Optionally, you can pass a dictionary
containing a mapping between strings and functions.
In the latter case, all functions are "assigned" to a string that is the function name,
except for function demo5 that we "assigned" to 'foo'.
All interactions with scheduler is done acting on the scheduler_* tables.
"""
docs.one_time = """
#### One time only
Okay, let's start with the tests....something simple: a function that needs to run one time only.
``
st.insert(task_name='one_time_only', function_name='demo4')
``:python
Instructions:
- Push "Clear All"
- Push "Start Monitoring"
- If not yet, start a worker in another shell ``web2py.py -K w2p_scheduler_tests``
- Wait a few seconds, a worker shows up
- Push "Queue Task"
- Wait a few seconds
What you should see:
- one worker is **ACTIVE**
- one scheduler_task gets **QUEUED**, goes into **RUNNING** for a while and then becomes **COMPLETED**
- when the task is **RUNNING**, a scheduler_run record pops up (**RUNNING**)
- When the task is **COMPLETED**, the scheduler_run record is updated to show a **COMPLETED** status.
Than, click "Stop Monitoring" and "Verify"
"""
comments.one_time = """
So, we got a task executed by scheduler, yeeeeahh!
Please note that you can get a lot of data to inspect execution in this mode
###### scheduler_task
- start_time is when you queued the task
- task_name is useful for retrieving the results later ``db(sr.scheduler_task.id == st.id)(st.task_name == 'one_time_only')(st.status == 'COMPLETED').select(sr.result, sr.output)``:python
- task gets a ``uuid`` by default
###### scheduler_run
- ``result`` is in json format
- ``output`` is the stdout, so you can watch your nice "print" statements
- ``start_time`` is when the task started
- ``stop_time`` is when the task stopped
- ``worker_name`` gets the worker name that processed the task
"""
docs.repeats = """
#### Repeating task
Let's say we want to run the demo1 function with some args and vars, 2 times.
``
st.insert(task_name="repeats", function_name='demo1', args=dumps(['a','b']), vars=dumps(dict(c=1, d=2)), repeats=2, period=30)
``
Instructions (same as before):
- Push "Clear All"
- Push "Start Monitoring"
- If not yet, start a worker in another shell ``web2py.py -K w2p_scheduler_tests``
- Wait a few seconds, a worker shows up
- Push "Queue Task"
- Wait a few seconds
Verify that:
- one worker is **ACTIVE**
- one scheduler_task gets **QUEUED**, goes into **RUNNING** for a while
- a scheduler_run record is created, goes **COMPLETED**
- task gets **QUEUED** again for a second round
- a new scheduler_run record is created
- task becomes **COMPLETED**
- a second scheduler_run record is created
Than, click "Stop Monitoring".
"""
comments.repeats = """
So, we got a task executed twice automatically, yeeeeahh!
###### scheduler_task
- times_run is 2
- last_run_time is when the second execution started
###### scheduler_run
- output args and vars got printed ok.
- start_time of the second execution is after ``period*seconds`` after start_time of the first_run
"""
docs.repeats_failed = """
#### Retry Failed
We want to run a function once, but allowing the function to raise an exception once.
That is, you want the function to "retry" an attempt if the first one fails.
We'll enqueue demo2, that we know if will fail in both runs, just to check if everything
works as expected (i.e. it gets re-queued only one time after the first FAILED run)
``
st.insert(task_name='retry_failed', function_name='demo2', retry_failed=1, period=30)
``
"""
docs.expire = """
#### Expired status
To better understand the use of ``stop_time`` parameter we're going to schedule
a function with stop_time < now. Task will have the status **QUEUED**, but as soon
as a worker see it, it will set its status to **EXPIRED**.
``
stop_time = request.now - datetime.timedelta(seconds=60)
st.insert(task_name='expire', function_name='demo4', stop_time=stop_time)
``
"""
docs.priority = """
#### Priority
Also if there is no explicit priority management for tasks you'd like to execute
a task putting that "on top of the list", for one-time-only tasks you can force the
``next_run_time`` parameter to something very far in the past (according to your preferences).
A task gets **ASSIGNED** to a worker, and the worker picks up (and execute) first tasks with
the minimum ``next_run_time`` in the set.
``
next_run_time = request.now - datetime.timedelta(seconds=60)
st.insert(task_name='priority1', function_name='demo1', args=dumps(['scheduled_first']))
st.insert(task_name='priority2', function_name='demo1', args=dumps(['scheduled_second']), next_run_time=next_run_time)
``
"""
docs.returns_null = """
#### Tasks with no return value
Sometimes you want a function to run, but you're not interested in the return value
(because you save it in another table, or you simply don't mind the results).
Well, there is no reason to have a record into the scheduler_run table!
So, by default, if a function doesn't return anything, its scheduler_run record
will be automatically deleted.
The record gets created anyway while the task is **RUNNING** because it's a way to
tell if a function is taking some time to be "executed", and because if task fails
(timeouts or exceptions) the record is needed to see what went wrong.
We'll queue 2 functions, both with no return values, demo3 that generates an exception
``
st.insert(task_name='no_returns1', function_name='demo5')
st.insert(task_name='no_returns2', function_name='demo3')
``
"""
docs.times_out = """
#### Tasks that will be terminated because they run too long
``
st.insert(task_name='timeout1', function_name='demo5', timeout=10, sync_output=2)
st.insert(task_name='timeout2', function_name='demo6', timeout=10, sync_output=2)
``
"""
return dict(docs=docs, comments=comments)
def workers():
steps = ['enabled', 'expiring', 'group_names_percentage',
'die_automatically', 'traceback', 'kill', 'terminate',
'disabled', 'return_values', 'discard']
docs = Storage()
comments = Storage()
docs.disabled = """
#### Disable a worker
A disabled worker won't pick any tasks at all, but as soon as its status is set to **ACTIVE**
again, it will start to process tasks.
Instructions:
- Push "Clear All"
- Push "Start Monitoring"
- If not yet, start a worker in another shell ``web2py.py -K w2p_scheduler_tests``
- Wait a few seconds, a worker shows up
- Push "Disable worker"
- Wait a few seconds
- Push "Queue task"
- see that the task remain in the **QUEUED** status
- Push "Activate worker"
- the task get **ASSIGNED** and processed a few seconds later
"""
docs.terminate = """
#### Terminate a worker
A worker with the status **TERMINATE** will die only if it's not processing any task.
Instructions:
- Push "Clear All"
- Push "Start Monitoring"
- If not yet, start a worker in another shell ``web2py.py -K w2p_scheduler_tests``
- Wait a few seconds, a worker shows up
- Push "Queue task"
- Wait until the task is reported as **RUNNING**
- Quick!!! Click on "Terminate worker"
- Watch the scheduler finish the current task and then die
"""
docs.kill = """
#### Kill a worker
A worker with the status **KILL** will die also if it's processing a task.
Instructions:
- Push "Clear All"
- Push "Start Monitoring"
- If not yet, start a worker in another shell ``web2py.py -K w2p_scheduler_tests``
- Wait a few seconds, a worker shows up
- Push "Queue task"
- Wait until the task is reported as **RUNNING**
- Quick!!! Click on "Kill worker"
- Watch the scheduler die within 3 seconds
"""
docs.times_out = """
#### Let the task timeout
As soon as the task runs longer than its timeout it is stopped and a full traceback is produced (if possible).
Instructions:
- Push "Clear All"
- Push "Start Monitoring"
- If not yet, start a worker in another shell ``web2py.py -K w2p_scheduler_tests``
- Wait a few seconds, a worker shows up
- Push "Queue task"
- Wait until the task is reported as **RUNNING**
- Watch the task to timeout within 10 seconds
"""
return dict(docs=docs, comments=comments)
