Move cleanup, cron tasks to the maintenance node

[sanjaysrikakulam]

Through this PR all cron tasks, cleanups etc that can be moved to the maintenance node are being moved.

ToDo Manully on sn06:

1. Role: usegalaxy-eu.galaxy-slurp
   1. Remove files:
      1. /usr/bin/galaxy-slurp
      2. /usr/bin/galaxy-slurp-upto
   2. Remove cron jobs from galaxy user:
      1. Slurp daily Galaxy stats into InfluxDB
      2. Slurp up-to-today galaxy stats into InfluxDB upto version
2. Role: usegalaxy-eu.htcondor_release
   1. Remove file:
      1. /usr/bin/htcondor-release-held-jobs
   2. Remove cron job from galaxy user:
      1. Condor release held jobs increasing memory
3. Role: usegalaxy-eu.fix-unscheduled-workflows
   1. Remove file:
      1. /usr/bin/galaxy-fix-unscheduled-workflows
   2. Remove cron job (if exists):
      1. Fix unscheduled workflows galaxyproject/galaxy#8209
4. Role: usegalaxy-eu.fix-ancient-ftp-data
   1. Remove files:
      1. /usr/bin/fix-ftp
   2. Remove cron jobs from galaxy user:
      1. Remove old FTP data
      2. Fix ftp
5. Role: usegalaxy-eu.fix-user-quotas
   1. Remove files:
      1. /usr/bin/galaxy-fix-user-quotas
      2. /usr/bin/galaxy-fix-elixir-user-quotas
   2. Remove cron jobs from galaxy user:
      1. Recalculate user quotas
      2. Attribute ELIXIR quota to ELIXIR AAI users
6. Role: usegalaxy-eu.fix-stop-ITs
   1. Remove files:
      1. /usr/bin/stop-ITs
   2. Remove cron jobs from galaxy user:
      1. Stop ITs older than 24h
   3. Remove log rotate:
      1. /etc/logrotate.d/stop-ITs
7. Role: usegalaxy_eu.fs_maintenance
   1. Remove cron jobs from galaxy user:
      1. Condor maintenance tasks submitter

Other dependencies:

1. Merge this PR: Change the fs-htcondor jobs universe to local ansible-fs-maintenance#4, and build a new release
2. Add the latest version to the requirements.yml file

This PR moves 6 roles from sn06 to the maintenance node and only some tasks from the usegalaxy_eu.fs_maintenance role (the gxadmin tasks in fsm_cron_tasks uses the galaxy's log directory /var/log/galaxy for cleanup, so it should stay on sn06).

The above cleanup should be performed for each role on sn06 so there will not be any redundancy.

[kysrpex]

I see that the variables removed from sn06 were already in maintenance.yml.

Thus, it looks good. I hope I did not miss anything.

[sanjaysrikakulam]

Other dependencies mentioned above are now done.

I will do the manual clean up now.

[sanjaysrikakulam]

Cleanup of files and the respective cronjobs are now removed.

Merging this PR.

[bgruening]

What does the fsm_cron_tasks do?

[sanjaysrikakulam]

What does the fsm_cron_tasks do?

fsm_cron_tasks runs gxadmin galaxy cleanup 60; this gxadmin needs access to the galaxy log files located in /var/log/galaxy. So unfortunately this needs to run on the headnode for now.