0.9.3

0.9.3 (2024-03-13)

✨ What's new in REANA 0.9.3

REANA 0.9.3 is a minor update that upgrades Snakemake workflow engine to version 7, improves job submission performance for massively-parallel workflows, improves the clean-up processing for stopped and failed workflows, and brings other minor improvements and bug fixes.

Please see the REANA 0.9.3 release blog post for more information.

⚡ Detailed changelog for REANA 0.9.3 components

reana 0.9.3 (2024-03-13)

• [Build] helm: add support for Kubernetes 1.29 (#775) (ae90500)
• [Features] helm: add value to customise env vars of job controller (#781) (634691f)
• [Features] helm: add value to customise PostgreSQL docker image (#774) (07a191f), closes #773
• [Features] helm: add value to customise URL of privacy notice (#778) (650ddbd)
• [Features] helm: add values to customise env vars of workflow engines (#781) (35ee032)
• [Features] helm: use PostgreSQL 14.10 in local dev deployment (#774) (43ead8a), closes #744
• [Features] reana-dev: add git-aggregate-changelog (#789) (6210b11)
• [Features] reana-dev: add date to commits bumping dependencies (#787) (a4cb84c)
• [Bug fixes] reana-dev: add PR number in commits bumping shared modules (#783) (57c6755)
• [Bug fixes] reana-dev: create commits that conform to conventional style (#777) (86d0133)
• [Bug fixes] reana-dev: delete extra files with git-submodule --update (#764) (e5680ce)
• [Bug fixes] reana-dev: detect number of already open PR in commit suffix (#783) (c533c34)
• [Bug fixes] reana-dev: update container image labels when releasing (#765) (fe6bc2c)
• [Code refactoring] docs: move from reST to Markdown (#776) (79aedb9)
• [Code style] black: format with black v24 (#772) (311e157)
• [Continuous integration] commitlint: addition of commit message linter (#767) (be77666)
• [Continuous integration] commitlint: allow release commit style (#785) (a6f95ac)
• [Continuous integration] commitlint: check for the presence of concrete PR number (#771) (2c34634)
• [Continuous integration] release-please: initial configuration (#767) (bb45539)
• [Continuous integration] release-please: update version in Helm Chart (#770) (09c9210)
• [Continuous integration] shellcheck: fix exit code propagation (#771) (035d51c)
• [Documentation] authors: complete list of contributors (#779) (123eae8)
• [Documentation] chat: remove Gitter (#782) (aba8ac2)

reana-client 0.9.3 (2024-03-13)

• [Build] appimage: upgrade to Python 3.8.18 (#704) (783c17a)
• [Bug fixes] status: display correct duration of stopped workflows (#701) (b53def8), closes #699
• [Code refactoring] docs: move from reST to Markdown (#703) (c9c4d53)
• [Code style] black: format with black v24 (#702) (02dc830)
• [Test suite] snakemake: allow running Snakemake 7 tests on Python 3.11+ (#700) (8ad7ff1), closes #655
• [Continuous integration] commitlint: addition of commit message linter (#695) (2de7d61)
• [Continuous integration] commitlint: allow release commit style (#708) (f552752)
• [Continuous integration] commitlint: check for the presence of concrete PR number (#698) (fa5b7c7)
• [Continuous integration] pytest: install tests package variant instead of all (#703) (fe0b00a)
• [Continuous integration] release-please: initial configuration (#695) (5b278f1)
• [Continuous integration] shellcheck: fix exit code propagation (#698) (fe696ea)
• [Documentation] authors: complete list of contributors (#705) (875997c)

reana-commons 0.9.8 (2024-03-01)

• [Build] python: change extra names to comply with PEP 685 (#446) (9dad6da)
• [Build] python: require smart-open<7 for Python 3.6 (#446) (17fd581)
• [Build] python: restore snakemake reports extra (#446) (904178f)
• [Continuous integration] commitlint: allow release commit style (#447) (1208ccf)

reana-commons [0.9.7](https://github.com/reanahub/reana-commons/co...

0.9.2

• Users:
  • Adds web interface form allowing to generate launcher URL for any user-provided public analysis, as well as the Markdown snippet for the corresponding Launch-on-REANA badge.
  • Adds web interface option to delete all the runs of a workflow.
  • Changes the Launch-on-REANA web interface page to improve how workflow parameters are shown by displaying them inside a table.
  • Changes CVMFS support to allow users to automatically mount any available repository.
  • Changes the REANA specification schema to use the draft-07 version of the JSON Schema specification.
  • Changes reana-client validate command to show detailed errors when the specification file is not a valid YAML file.
  • Changes validation of REANA specification to make the environment property mandatory for the steps of serial workflows.
  • Changes validation of REANA specification to raise a warning for unexpected properties for the steps of serial workflows.
  • Changes validation of REANA specification to report improved validation warnings which also indicate where unexpected properties are located in the file.
  • Changes workflow restarts to allow for more than nine restarts of the same workflows.
  • Changes workflow scheduler logging behaviour to also report the main reason behind scheduling errors to the users.
  • Fixes reana-client list command to accept case-insensitive column names when sorting the returned workflow runs via the --sort option.
  • Fixes reana-client run wrapper command for workflows that do not contain inputs clause in their specification.
  • Fixes reana-client's create_workflow_from_json API function to always load and send the workflow specification to the server.
  • Fixes creation of image thumbnails for output files in Snakemake HTML execution reports.
• Administrators:
  • Changes several database index definitions in order to improve performance of most common database queries.
  • Changes the names of database table, column, index and key constraints in order to follow the SQLAlchemy upstream naming conventions everywhere.
  • Changes the Workflow table to replace the run_number column with two new columns run_number_major and run_number_minor in order to allow for more than nine restarts of user workflows.
  • Changes CVMFS support to allow users to automatically mount any available repository, thanks to CVMFS CSI v2.
  • Fixes the mounting of CVMFS volumes for the REANA deployments that use non-default Kubernetes namespace.
  • Fixes container image building of cluster components for the arm64 architecture.
  • Fixes job monitoring in cases when job creation fails, for example when it is not possible to successfully mount volumes.
  • Fixes job status consumer exception while attempting to fetch workflow engine logs for workflows that could not have been successfully scheduled.
  • Fixes the creation of Kubernetes jobs by retrying in case of error and by correctly handling the error after reaching the retry limit.
• Developers:
  • Adds automated multi-platform container image building of cluster components for amd64 and arm64 architectures.
  • Adds new --image-name option to the reana-dev docker-push command to customise the name of the docker image to publish.
  • Adds new --platform option to the reana-dev docker-build and reana-dev release-docker commands to build and publish multi-platform images.
  • Adds new --registry option to the reana-dev docker-push and reana-dev release-docker commands to specify the registry where the built docker images should be pushed to.
  • Adds new --tags-only option to the reana-dev release-docker command to only print the final docker image names, without pushing the images to the registry.
  • Adds new reana-dev git-create-release-branch command to create a new Git branch for a new release.
  • Adds new reana-dev git-upgrade-requirements command to upgrade the requirements.txt file before a new release.
  • Changes reana-dev git-fork and reana-dev git-create-pr to use the gh CLI client instead of hub.
  • Changes reana-dev python-run-tests command to allow excluding certain Python components.
  • Changes reana-dev python-run-tests command to allow execution of selected pytests only by passing over PYTEST_ADDOPTS environment variable.
  • Changes validation of REANA specification to expose functions for loading workflow input parameters and workflow specifications.
  • Changes version of reana-ui Node.js Docker image from 16 to 18.
  • Changes the workflow deletion endpoint to return a different and more appropriate message when deleting all the runs of a workflow.
  • Changes the workflow list endpoint on how pagination is performed in order to avoid counting twice the total number of records.
  • Fixes reana-dev python-run-tests command to create Python-3.8 based virtual environments to use the same version as container images.

0.9.1

• Users:
  • Adds support for Python 3.12.
  • Adds support for previewing PDF files present in a workflow's workspace.
  • Adds support for previewing ROOT files present in a workflow's workspace.
  • Adds support for signing-in with a custom third-party Keycloak instance.
  • Adds a new menu item to the workflow actions popup to allow stopping running workflows.
  • Adds prune command to delete all intermediate files of a given workflow. Use with care.
  • Changes the deletion of a workflow to automatically delete an open interactive session attached to its workspace.
  • Changes the workflow deletion message to clarify that attached interactive sessions are also closed when a workflow is deleted.
  • Changes the workflow progress bar to always display it as completed for finished workflows.
  • Changes the interactive session notification to also report that the session will be closed after a specified number of days of inactivity.
  • Changes the workflow-details page to make it possible to scroll through the list of workflow steps in the job logs section.
  • Changes the workflow-details page to not automatically refresh the selected job when viewing the related logs, but keeping the user-selected one active.
  • Changes the page titles to conform to the same sentence case style.
  • Changes the launcher page to show warnings when validating the REANA specification file of the workflow to be launched.
  • Changes open command to inform user about the auto-closure of interactive sessions after a certain inactivity timeout.
  • Changes validate command to display non-critical validation warnings when checking the REANA specification file.
  • Changes Rucio authentication helper to allow users to override the Rucio server and authentication hosts independently of VO name.
  • Fixes job status inconsistency when stopping a workflow by setting the job statuses to stopped for any running jobs.
  • Fixes calculation of workflow runtime durations for stopped workflows.
  • Fixes list command to correctly list workflows when sorting them by their run number or by the size of their workspace.
  • Fixes du command help message typo.
  • Fixes validation --environments command to correctly handle fully-qualified image names.
  • Fixes deletion of failed jobs not being performed when Kerberos is enabled.
  • Fixes job monitoring to consider OOM-killed jobs as failed.
  • Fixes detection of default Rucio server and authentication host for ATLAS VO.
  • Fixes the reported total number of jobs for restarted workflows by excluding cached jobs that were simply reused from previous runs in the workspace and not really executed by Snakemake.
  • Fixes an issue where Snakemake workflows could get stuck waiting for already-finished jobs.
• Administrators:
  • Adds support for Kubernetes clusters 1.26, 1.27, 1.28.
  • Adds new configuration option components.reana_ui.launcher_examples to customise the demo examples that are shown in the launch page in the REANA UI.
  • Adds new configuration option interactive_sessions.maximum_inactivity_period to set a limit in days for the maximum inactivity period of interactive sessions after which they will be closed.
  • Adds new configuration option interactive_sessions.cronjob_schedule to set how often interactive session cleanup should be performed.
  • Adds new configuration option ingress.extra to define extra Ingress resources, in order to support redirecting HTTP requests to HTTPS with traefik v2 version.
  • Adds new configuration option ingress.tls.hosts to define hosts that are present in the TLS certificate, in order to support cert-manager's automatic creation of certificates.
  • Adds new configuration option notifications.email_config.smtp_ssl to use SSL when connecting to the SMTP email server.
  • Adds new configuration option notifications.email_config.smtp_starttls to use the STARTTLS command to enable encryption after connecting to the SMTP email server.
  • Adds new configuration option components.reana_ui.file_preview_size_limit to set the maximum file size that can be previewed in the web interface.
  • Adds new configuration options login and secrets.login for configuring Keycloak SSO login with third-party authentication services.
  • Adds new interactive-session-cleanup command that can be used by REANA administrators to close interactive sessions that are inactive for more than the specified number of days.
  • Adds progress meter to the logs of the periodic quota updater.
  • Adds the content of the secrets.gitlab.REANA_GITLAB_HOST configuration option to the list of GitLab instances from which it is possible to launch a workflow.
  • Changes uWSGI configuration to increase buffer size, add vacuum option, etc.
  • Changes CPU and disk quota calculations to improve the performance of periodic quota updater.
  • Changes the system status report to simplify and clarify the disk usage summary.
  • Changes check-workflows command to also check the presence of workspaces on the shared volume.
  • Changes check-workflows command to not show in-sync runs by default. If needed, they can be shown using the new --show-all option.
  • Changes reana-admin command options to require the passing of --admin-access-token argument more globally.
  • Changes the k8s specification for interactive session pods to include labels for improved subset selection of objects.
  • Changes the k8s specification for interactive session ingress resource to include annotations.
  • Changes nginx configuration to save bandwidth by serving gzip-compressed static files.
  • Changes HTCondor to version 9.0.17 (LTS).
  • Fixes uWSGI memory consumption on systems with very high allowed number of open files.
  • Fixes cronjob failures due to database connection issues when REANA is deployed with non-default namespace or prefix.
  • Fixes ingress.enabled option to correctly enable or disable the creation of Ingresses.
  • Fixes graceful shutdown for reana-server and reana-workflow-controller.
  • Fixes the workflow priority calculation to avoid workflows stuck in the queued status when the number of allowed concurrent workflow is set to zero.
  • Fixes GitLab integration to automatically redirect the user to the correct URL when the access request is accepted.
  • Fixes authentication flow to correctly deny access to past revoked tokens in case the same user has also other new active tokens.
  • Fixes email templates to show the correct kubectl commands when REANA is deployed inside a non-default namespace or with a custom component name prefix.
  • Fixes email sender for system emails to notifications.email_config.sender Helm value.
  • Fixes email receiver for token request emails to use notifications.email_config.receiver Helm value.
  • Fixes quota-set-default-limits command to propagate default quota limits to all users without custom quota limit values.
  • Fixes job status consumer to correctly rollback the database transaction when an error occurs.
  • Fixes intermittent Slurm connection issues by DNS-resolving the Slurm head node IPv4 address before establishing connections.
  • Fixes Slurm command generation issues when using fully-qualified image names.
  • Fixes high memory usage of RabbitMQ by limiting the maximum number of open file descriptors.
  • Removes support for Kubernetes version prior to 1.21.
• Developers:
  • Adds new prune_workspace endpoint to allow users to delete all the files of a workflow, specifying whether to also delete the inputs and/or the outputs.
  • Adds the timestamp of when the workflow was stopped (run_stopped_at) to the workflow list and the workflow status endpoints.
  • Adds unique error messages to Kubernetes job monitor to more easily identify source of problems.
  • Adds new --parallel option to docker-build, cluster-build and run-ci to build multiple docker images in parallel.
  • Changes launch endpoint to also include the warnings of the validation of the workflow specification.
  • Changes OpenAPI specification of the info endpoint to return the maximum inactivity time before automatic closure of interactive sessions.
  • Changes apispec dependency version in order to be compatible with PyYAML v6.
  • Changes Paramiko to version 3.0.0.
  • Changes remote storage file support for Snakemake workflows to use XRootD 5.6.0.
  • Fixes cluster-deploy, cluster-undeploy and client-setup-environment commands when using custom instance name or kubernetes namespace.
  • Fixes the git-tag command to display the component name.
  • Fixes container image names to be Podman-compatible.
  • Fixes location of HTCondor build dependencies.

0.9.1-alpha.4

REANA reproducible research data analysis platform

0.9.1-alpha.3

REANA reproducible research data analysis platform

0.9.1-alpha.2

REANA reproducible research data analysis platform

0.9.1-alpha.1

REANA reproducible research data analysis platform

0.9.0

• Users:
  • Adds support for Rucio authentication for workflow jobs.
  • Adds support for Kerberos authentication for workflow orchestration.
  • Adds support for specifying slurm_partition and slurm_time for Slurm compute backend jobs.
  • Adds support for XRootD remote file locations in Snakemake workflow specification definitions.
  • Adds support for Python 3.11.
  • Adds Launch on REANA page allowing the submission of workflows via badge-clicking.
  • Adds notifications to inform users when critical levels of quota usage is reached.
  • Adds 404 Not Found error page.
  • Adds tab titles to all the pages.
  • Adds the REANA_WORKSPACE environment variable to jupyter notebooks and terminals.
  • Adds option to sort workflows by most disk and cpu quota usage to the workflow list endpoint.
  • Adds support for specifying and listing workspace file retention rules.
  • Adds support for .gitignore and .reanaignore to specify files that should not be uploaded to REANA.
  • Adds retention-rules-list command to list the retention rules of a workflow.
  • Changes REANA specification to allow specifying retention_days for the workflow.
  • Changes default Slurm partition to inf-short.
  • Changes GitLab integration to also retrieve user's projects that are in groups and subgroups.
  • Changes the workflow-details page to show the logs of the workflow engine.
  • Changes the workflow-details page to show file sizes in a human-readable format.
  • Changes the workflow-details page to show the workspace's retention rules.
  • Changes the workflow-details page to show the duration of the workflow's jobs.
  • Changes the workflow-details page to display a label of the workflow launcher URL remote origin.
  • Changes the workflow-details page to periodically refresh the content of the page.
  • Changes the workflow-details page to refresh after the deletion of a workflow.
  • Changes the workflow-list page to add a way to hide deleted workflows.
  • Changes the workflow-list page to add new workflows sorting options by most used disk and cpu quota.
  • Changes the deletion of a workflow to always clean up the workspace and to update the user disk quota usage.
  • Changes the CWD of jupyter's terminals to the directory of the workflow's workspace.
  • Changes percentage ranges used to calculate the health status of user resource quota usage.
  • Changes create and restart commands to always upload the REANA specification file.
  • Changes delete command to always delete the workflow's workspace.
  • Changes delete_workflow Python API function to always delete the workflow's workspace.
  • Changes download command to add the possibility to write files to the standard output via -o - option.
  • Changes list command to hide deleted workflows by default.
  • Changes list command to allow displaying deleted workflows via --all and --show-deleted-runs options.
  • Changes list and status commands to allow displaying the duration of workflows with the --include-duration option.
  • Changes mv command to allow moving files while a workflow is running.
  • Changes upload command to prevent uploading symlinks.
  • Changes validation --environment command to use Docker registry v2 APIs to check that a Docker image exists in DockerHub.
  • Fixes list command to highlight the workflow specified in REANA_WORKON correctly.
  • Fixes secrets-delete command error message when deleting non existing secrets.
  • Fixes start command to report failed workflows as errors.
  • Fixes start and run commands to correctly follow the execution of the workflow until termination.
  • Fixes status command to respect output format provided by the --format option.
  • Fixes upload command to report when input directories are listed under the files section in the REANA specification file and vice versa.
  • Fixes validate --environment command to detect illegal whitespace characters in Docker image names.
  • Fixes Kerberos authentication for long-running workflows by renewing the Kerberos ticket periodically.
  • Fixes status reporting for failed CWL and Snakemake jobs that were incorrectly considered successful.
  • Fixes redirection chain for non-signed-in CERN SSO users to access the desired target page after sign-in.
  • Fixes the ordering by size of the files showed in the Workspace tab of the workflow-details page.
  • Fixes CERN OIDC authentication to possibly allow eduGAIN and social login users.
  • Fixes wrong numbering of restarted workflows by limiting the number of times a workflow can be restarted to nine.
• Administrators:
  • Adds new configuration environment variable reana_server.environment.REANA_SCHEDULER_REQUEUE_COUNT to set workflow requeue count in case of scheduling errors or busy cluster situations.
  • Adds "infinity" option to REANA_SCHEDULER_REQUEUE_COUNT to disable requeue count.
  • Adds support for Kubernetes clusters 1.22, 1.23, 1.24, 1.25.
  • Adds new configuration option workspaces.retention_rules.maximum_period to set a default period for workspace retention rules.
  • Adds new configuration option workspaces.retention_rules.cronjob_schedule to set how often pending retention rules should be applied.
  • Adds configuration environment variable reana_server.environment.REANA_RATELIMIT_SLOW to limit API requests to some protected endpoints e.g launch workflow.
  • Adds configuration environment variable reana_server.environment.REANA_WORKFLOW_SCHEDULING_READINESS_CHECK_LEVEL to define checks that are performed to assess whether the cluster is ready to start new workflows.
  • Adds new configuration option ingress.tls.self_signed_cert to enable the generation of a self-signed TLS certificate.
  • Adds new configuration option ingress.tls.secret_name to specify the name of the Kubernetes secret containing the TLS certificate to be used.
  • Adds support for configuring an additional volume to be used by the database and the message broker.
  • Adds new configuration option maintenance.enabled to scale down the cluster for maintenance.
  • Adds support for Unicode characters inside email body.
  • Adds queue-consume command that can be used by REANA administrators to remove specific messages from the queue.
  • Adds retention-rules-apply command that can be used by REANA administrators to apply pending retention rules.
  • Adds retention-rules-extend command that can be used by REANA administrators to extend the duration of active retentions rules.
  • Adds check-workflows command that can be used by REANA administrators to check for out-of-sync workflows and interactive sessions.
  • Changes configuration option quota.workflow_termination_update_policy to deactivate workflow termination accounting by default.
  • Changes Helm template to use PostgreSQL 12.13 version.
  • Changes the base image for most of the components to Ubuntu 20.04 LTS and reduces final Docker image size by removing build-time dependencies.
  • Changes reana-auth-vomsproxy sidecar to the latest stable version to support client-side proxy file generation technique and ESCAPE VOMS.
  • Changes OAuth configuration to enable the new CERN SSO.
  • Changes job status consumer to improve logging for not-alive workflows.
  • Changes the deployment of interactive sessions to improve security by not automounting the Kubernetes service account token.
  • Changes the deployment of job-controller to avoid unnecessarily mounting the database's directory.
  • Changes the announcements to support limited HTML markup.
  • Changes REANA specification loading functionality to allow specifying different working directories.
  • Changes global setting of maximum number of parallel jobs to 300 for Snakemake workflow engine.
  • Fixes job status consumer by discarding invalid job IDs.
  • Fixes GitLab integration error reporting in case user exceeds CPU or Disk quota usage limits.
  • Fixes issue when irregular number formats are passed to REANA_SCHEDULER_REQUEUE_COUNT configuration environment variable.
  • Fixes quota updater to reduce memory usage.
  • Fixes conversion of possibly-negative resource usage values to human-readable formats.
  • Fixes disk quota updater to prevent setting negative disk quota usage values.
  • Removes support for Kubernetes version prior to 1.19.
• Developers:
  • Adds OpenAPI specification support for launch endpoint that allows running workflows from remote sources.
  • Adds OpenAPI specification support for get_workflow_retention_rules endpoint that allows to retrieve the workspace file retention rules of a workflow.
  • Adds the remote origin of workflows submitted via Launch-on-REANA (launcher_url) to the workflow list endpoint.
  • Adds common utility functions for managing workspace files to reana-commons.
  • Changes default consumer prefetch count to handle 10 messages instead of 200 in order to reduce the probability of 406 PRECONDITION errors on message acknowledgement.
  • Changes git-upgrade-shared-modules to generate the correct upper-bound in setup.py.
  • Changes REANA specification loading and validation functionalities by porting some of the logic to reana-commons.
  • Changes OpenAPI specification to include missing response schema elements.
  • Changes the Kubernetes Python client to use the networking/v1 API.
  • Changes the deployment of interactive sessions to use networking/v1 Kubernetes API.
  • Changes to Flask v2.
  • Changes /api/info endpoint to also include the kubernetes maximum memory...

0.9.0-alpha.8

REANA reproducible research data analysis platform

0.9.0-alpha.7

REANA reproducible research data analysis platform