A collection of Python utilities.

Use a virtualenv for development. For example, install kobo and dependencies in editable mode:

virtualenv ~/kobo-dev
. ~/kobo-dev/bin/activate
pip install --editable ~/src/kobo
pip install -rtest-requirements.txt

To run the test suite:

• Run py.test to run tests against installed versions of python and dependencies, or...
• Install and run tox to run test suite against a matrix of supported python and Django versions (more thorough, slower).

Please submit pull requests against https://github.com/release-engineering/kobo.

• hub: fixed display of permissions in user details template
• worker: fixed a logging deadlock introduced in kobo 0.36.0
• worker: fixed a locked TaskManager sometimes opening tasks

• worker: on SIGHUP, will now gracefully await task completion and re-exec

• client: the --hub option to watch-log command is no longer ignored
• worker: logging thread is more robust, recovers from many errors
• hub: minor fixes to worker detail UI
• hub: respond with 404 error on missing logs, rather than 500
• other minor fixes

• client: fixed support for proxies (https_proxy) in XML-RPC client

• Support Django 5.0

• hub: restricted login redirects to same site only
• hub: fixed assignment of tasks to workers with incompatible arch
• hub: fixed display of arches/channels on worker detail page
• hub: avoid SELinux relabelling issue during file upload
• Minor cleanups of obsolete or deprecated code

• hub: the / character may now be used within usernames

• hub: uploads of the same name but different content are no longer considered duplicates

• hub: added a page for listing failed tasks, alongside the existing Running/Finished task lists.

• hub: introduced inherit_worker argument to create_subtask API, allowing workers to ensure created subtasks are assigned to themselves

• hub: introduced settings.USERS_ACL_PERMISSION to adjust access to user list

• hub: minor fixes to page titles

• hub: deleting a worker no longer deletes associated tasks

• hub: decorate Task.save with transaction.atomic
• hub: do not call save twice in Task.save()
• hub: fix Task.cancel_subtasks
• client: allow file upload size to be sent a string
• django: use BigIntegerField for file upload sizes
• hub: return error 404 for non-existent logs
• hub: move filename resolution to _streamed_log_response
• worker: do not repeatedly remove already awaited tasks from subtask list
• django: reimplement django_version_ge in pure Python
• hub: do not use workers without ids in a filter
• hub: replace pipes with shlex

• Make xmlrpc timeout configurable
• django: remove support for Django 1.10 and older
• hub: update Worker.update_worker documentation

• Fixed XML-RPC authentication regression introduced in kobo 0.29.0

• Introduced create-worker command
• Improved compatibility with Django 4.2
• Improved output from admin commands on XML-RPC errors
• Cleaned up unused code

• Renamed deprecated get_query_set to get_queryset
• Changed repr to properly deserialize tracebacks sent over XML-RPC
• Fixed remaining Django 3 deprecations
• Fixed reporting of failed uploads
• Changed behavior that log decoding failure doesn't raise an error
• Made it possible to interrupt assigned tasks

• Retired python 2 support
• Added Client Credentials Flow OIDC authentication

• Added logging for authN/authZ events
• Added --hub option to client

• Fixed calculating the number of task's subtasks in new Django versions
• Fixed cli/help_RST compatibility with Python 3
• fixed encoding errors by forcing the usage of UTF-8 in str and bytes functions

• Added an option to make subtask inherit parents priority and make it configurable

• Improved compatibility with Django 4.1
• Improved compatibility with python 3

• Fixed a crash from Django dumpdata command when using Django 2 and later.

• Fixed an encoding error when using a python 2 XML-RPC client against a python 3 kobo hub.

• It is now possible to provide a custom task manager implementation into the kobo worker's main loop. This can be used to alter the worker's task scheduling or execution behavior.

• Introduced AUTH_METHOD=oidc for XML-RPC clients. If this authentication method is enabled, the OIDC endpoint introduced by kobo 0.22.0 will be used.
• HubProxy now respects the system configuration for kerberos realm when KRB_REALM has not been set.

• hub: added an endpoint for OIDC login. Note that kobo itself does not implement any OIDC workflow, which may instead be implemented by the hosting web server.
• HubProxy will now retry on additional cases of XML-RPC error.

• Fixed a crash bug in LoggingThread when mixing bytes and text.

• Added methods to HubProxy for explicitly logging into hub.

• Many compatibility fixes for modern Python and Django versions.
• Fixed loss of error details when LoggingThread fails.

• Improved Django 4.x compatibility

• kobo.rpmlib.get_keys_from_header now correctly handles RPMs using header-only signing. Also fixes pkgset.SimpleRpmWrapper.signature.

• kobo workers now maintain a last_seen field which may be used to monitor the availability of workers.

• The resubmit-tasks now supports adjusting the priority of the resubmitted task. The client and server must both upgrade to kobo 0.19.0 in order to use this feature.

• Fixed backwards-incompatible change in kobo 0.18.0: if a kobo-derived client provided its own --profile argument, this could clash with the new argument in kobo.

• The kobo client now supports a --profile argument to select an alternative configuration file. To enable this functionality, users of kobo should provide a default_profile and configuration_directory when creating a CommandOptionParser.

• Fixed a python3 compatibility issue in kobo.http.
• Fixed a crash bug for projects using the default XMLRPC_TEMPLATE for the XML-RPC help page with Django > 1.

• Introduced the concept of global variables to kobo.conf.PyConfigParser. This feature allows a variable to be declared in one file and overridden in a subsequently imported file.

• The client.get_worker_info XML-RPC method no longer requires authentication. It was formerly restricted to superusers only.

• The HubProxy auto_logout feature was removed. This feature was unreliable and could occasionally result in deadlocks, due to the usage of finalizers. It is recommended to call logout() from a finally block in cases where it's important to ensure a logout.

• Fixed a Django 1.x compatibility issue introduced in kobo 0.15.0

• Improved Django 3.x compatibility

• Introduced gssapi login method, for improved Python3 compatibility. This login method requires the hub's krb5login endpoint to be enabled. The default login method is unchanged.

• Fixed a Python3 compatibility issue in hub migrations
• Improved Django 1.11 compatibility

• Improved Django 1.11 and 2.x compatibility

• The resubmit-tasks command now accepts a --nowait argument

• Fixed usage of shortcuts.run with text mode in Python 3 (#133)
• Fixed decoding crashes in shortcuts.run when an incomplete multibyte sequence is read (#119)
• Fixed rpmlib attempting to decode binary data in RPM headers
• Fixed various Python 3 compatibility issues

• Improved Python 3 compatibility, particularly in xmlrpc and rpmlib modules
• Improved Django 1.11 and 2.x compatibility

• Testing of Python 2.6 has been dropped; some kobo functionality still works with Python 2.6, but may break without warning in later releases
• Support for Django 1.6 has been removed

• Fixed LoggingThread on Python 3 (#66)
• Fixed kobo.django.xmlrpc migrations for Django 2.x
• Fixed some exceptions discarded without logging (#32)
• Fixed some reliability issues in kobo.xmlrpc

• Improved Python 3 compatibility
• Improved Django 2.0 compatibility
• Improved tests coverage
• Header produced by kobo.shortcuts.run(show_cmd=True) is now limited to 79 characters length

• Fixed handling of string SERVER_PORT in wsgi requests
• Fixed Worker.timeout_task wrongly setting subtasks to INTERRUPTED (#72)
• Fixed Worker.set_task_weight always crashing (#75)

• Improved Python 3 compatibility
• Improved tests coverage

• kobo worker name no longer needs to match host FQDN
• improved error reporting when loading configuration
• improved error reporting from kobo.shortcuts.run
• reduced memory usage when handling large log files
• models now respect settings.AUTH_USER_MODEL

• fixed crash on xml-rpc client in python <= 2.7.9
• fixed spurious whitespace from kobo.shortcuts.run (#40)
• fixed missing migration for User model

• kobo.shortcuts.run now supports all Popen keyword arguments
• resubmit-tasks has a --force argument to resubmit successful tasks
• new watch-log command for watching a log from CLI
• admin UI now covers user model

• kobo.shortcuts.run now resumes on interrupted system calls
• worker load no longer includes assigned but unstarted tasks

• pkgset.SimpleRpmWrapper has now checksum_type member
• threads.run_in_threads helper function

Django part of kobo was udpated to be compatible with 1.5 release. Lower version are no more supported. As a side-effect, only python 2.6+ is supported by django part.

From same reason there will be no builds of kobo-django package for RHEL 5 and lower as it lacks required python version.

• RemoteUserMiddleware is now used for KrbV authentication
• LimitedRemoreUserMiddleware can be used to authenticate only on entry pages.
• LongnameUser model is used for auth backend. It allows 255 characters long user names (as they come from KrbV)
• Class-based generic views ExtraListView and ExtraDetailView. object_list helper stays for compatibility, but it is deprecated now and will be removed in future.

• State machine implementation - StateEnum, db field, form fields
• Brand new HTML template, media, views, urls and menu
• Menu supports django.root
• Task logs, javascript log watcher, threaded worker stdout logger
• XML-RPC help pages display list of contents
• Kerberos support in CookieTransport
• JSONField to store dicts and lists in database
• Add relative_path() and split_path() functions to shortcuts

file_name attribute renamed to file_path. file_name is now a property which returns actual file name.

Action: Change file_name to file_path in your code.

Username hack is enabled by default now (when kobo.django.auth is used). It changes username to 255 characters and also overrides validation RE.

Action: On postgresql run: ALTER TABLE auth_user ALTER username TYPE VARCHAR(255); Sqlite users have to use db_update-0.2.0-0.3.0 script.

Each worker's name must match it's FQDN now. This prevents cut&paste configuration errors when tasks end in INTERRUPTED state.

Action: Change worker names to FQDN. Change related usernames as well.

Removed 'lower_case' attribute. Plugins are now subclassed when a container is created. Each plugin now contains 'container' attribute.

Action: Remove 'lower_case' attribute from plugins, do whatever is necessary in 'normalize_name() class method instead.

• Improve PluginContainer inheritance. Also add 'container' attribute to each plugin class obtained from a container instance. (Daniel Mach)
• Plugins are now subclassed when a container is created. (Daniel Mach)

Field 'traceback' moved to a file (traceback.log). Field 'result' content dumped to a file (stdout.log), it is supposed to contain actual task result. Field 'args' changed to JSONField and data is directly available without any conversion.

Action: Run db_update-0.2.0-0.3.0 script.

Configuration no longer uses os.environ to get config file path. ClientCommandContainer, HubProxy and TaskManager constructor has a new mandatory 'conf' argument.

Action: config_file = os.environ.get("<PROJECT_NAME>_CONFIG_FILE", "/etc/<project_name>.conf") conf = kobo.conf.PyConfigParser() conf.load_from_file(config_file) ... and pass conf to ClientCommandContainer, HubProxy or TaskManager

TaskBase constructor has now a mandatory argument 'conf', which is automatically set in TaskManager.

Action: Add 'conf' argument to tasks classes with custom constructor.