Skip to content

Latest commit

 

History

History
315 lines (241 loc) · 13.2 KB

README.md

File metadata and controls

315 lines (241 loc) · 13.2 KB
Raw

django-huey-monitor

Django based tool for monitoring huey task queue

Current implementation will just store all Huey task signals into the database and display them in the Django admin.

tests codecov django-huey-monitor @ PyPi Python Versions License GPL-3.0-or-later

Quickstart

pip install django-huey-monitor
INSTALLED_APPS = [
    #...
    'bx_django_utils', # https://github.com/boxine/bx_django_utils
    'huey_monitor',
    #...
]

Note: You must normally not change your Huey tasks.

Collect main-/sub-tasks

Huey monitor model can store information about task hierarchy. But this information can't be collected automatically. You have to store these information in your Task code.

e.g.:

@task(context=True)
def sub_task(task, parent_task_id, chunk_data):
    # Save the task hierarchy by:
    TaskModel.objects.set_parent_task(
        main_task_id=parent_task_id,
        sub_task_id=task.id,
    )
    # ... do something with e.g.: chunk_data ...


@task(context=True)
def main_task(task):
    for chunk_data in something:
        sub_task(parent_task_id=task.id, chunk_data=chunk_data)

Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py

Collect progress information

Running task can store progress information in a similar way as tqdm. So it's possible to see the progress in admin.

Minimal example:

@task(context=True)
def foobar_task(task, list_to_process):
    process_info = ProcessInfo(
        task,
        desc='A description of this Job',
        total=len(list_to_process)
    )

    for item in list_to_process:
        # ...to something with the {item}...
        process_info.update(n=1) # add the information that one item was processed

It is also possible to divide the work to several tasks and collect information about the processing of main-/sub-tasks.

Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py

settings

override list filter (optional)

It is possible to override list_filter of SignalInfoModelAdmin and TaskModelAdmin via settings. e.g.:

HUEY_MONITOR_SIGNAL_INFO_MODEL_LIST_FILTER = ('task__name', 'signal_name')
HUEY_MONITOR_TASK_MODEL_LIST_FILTER = ('name', 'state__signal_name')

Note: This both settings are optional. In this example is the "hostname" filter not present ;)

run test project

Note: You can quickly test Huey Monitor with the test project, e.g:

~/django-huey-monitor$ ./manage.sh run_testserver

or in an isolated Docker container:

~/django-huey-monitor$ make up

More info see below.

Screenshots

(More Screenshots here: boxine.github.io/django-huey-monitor/)

2021-02-22-v030-task-details.png

2021-02-22-v030-task-details.png

2021-02-22-v030-progress-info1.png

2021-02-22-v030-progress-info1.png

developing

  • install docker
  • clone the project
  • start the container

To start developing e.g.:

~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py
~/django-huey-monitor$ make help
~/django-huey-monitor$ make up

Point our browser to: http://localhost:8000/

Our Makefile contains the following targets:

help                           List all commands
install                        install huey monitor package
update                         Update the dependencies as according to the pyproject.toml file
run-dev-server                 Run Django's developer server
test                           Run unittests
tox                            Run unittests via tox
makemessages                   Make and compile locales message files
clean                          Remove created files from the test project (e.g.: SQlite, static files)
build                          Update/Build docker services
up                             Start docker containers
down                           Stop all containers
shell-django                   go into a interactive bash shell in Django container
run-shell-django               Build and start the Django container and go into shell
shell-huey1                    go into a interactive bash shell in Huey worker container 1
shell-huey2                    go into a interactive bash shell in Huey worker container 2
shell-huey3                    go into a interactive bash shell in Huey worker container 3
shell-redis                    go into a interactive bash shell in Redis container
logs                           Display and follow docker logs
logs-django                    Display and follow docker logs only from "django" container
reload-django                  Reload the Django dev server
reload-huey                    Reload the Huey worker
restart                        Restart the containers
fire-test-tasks                Call "fire-test-tasks" manage command to create some Huey Tasks
fire-many-test-tasks           Call "fire-test-tasks" with --count 10000 to create many task entries ;)
fire-parallel-processing-task  Just fire "parallel processing" Huey Task
delete-all-tasks-data          Delete all Task/Signal database enties

It's also possible to run the test setup with SQLite and Huey immediate setup without docker:

~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py run_dev_server

Backwards-incompatible changes

Version compatibility

Huey Monitor Django Python
>0.10.0 v4.2, v5.0, v5.1 v3.11, v3.12
>v0.7.0 v3.2, v4.1, v4.2 v3.9, v3.10, v3.11
>v0.6.0 v3.2, v4.0, v4.1 v3.9, v3.10, v3.11
>v0.5.0 v2.2, v3.1, v3.2 v3.7, v3.8, v3.9
<=v0.4.0 v2.2, v3.0, v3.1 v3.7, v3.8, v3.9

v0.10.0

Set min. Python to v3.11. Remove Django 3.2.x and add Django v5.1.x to text matrix.

v0.6.0

We refactor the project setup: Developer must reinit the repository.

v0.5.0

Change CI and remove tests against Django 3.0, but add test run with Django v3.2

v0.3.0 -> v0.4.0 - Outsourcing Django stuff

bx_py_utils was split and Django related stuff moved into: bx_django_utils

You must change your Django settings and replace the app name:

 INSTALLED_APPS = [
     #...
-     'bx_py_utils',
+     'bx_django_utils',
     'huey_monitor',
     #...
 ]

History

License

GPL. Patches welcome!

Links