Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overview page modules #532

Merged
merged 43 commits into from
Aug 14, 2023
Merged

Overview page modules #532

merged 43 commits into from
Aug 14, 2023

Conversation

milachae
Copy link
Contributor

@milachae milachae commented Aug 7, 2023

#553

In this PR:

  • The script can be found at "scripts/module_list/module_overview.py". This generates an overview of all possible packages on the different clusters.
  • in the "scripts/module_list/tests", tests can be found. Most of the lines in this PR are test files, not actual code. Only +- 175 lines of actual.
  • A new workflow has been added to test/lint the script

scripts/module_list/tests/data/data_avail_simple.txt Outdated Show resolved Hide resolved
scripts/module_list/tests/test_data.py Outdated Show resolved Hide resolved
scripts/module_list/tests/test_data.py Outdated Show resolved Hide resolved
scripts/module_list/tests/test_module.py Outdated Show resolved Hide resolved
.github/workflows/script_module_list.yml Outdated Show resolved Hide resolved
.github/workflows/script_module_list.yml Outdated Show resolved Hide resolved
scripts/README.md Outdated Show resolved Hide resolved
scripts/module_list/README.md Outdated Show resolved Hide resolved
scripts/module_list/module_overview.py Outdated Show resolved Hide resolved
scripts/module_list/module_overview.py Outdated Show resolved Hide resolved
@param filter_fn: Filter function on the ouput.
@return:
"""
lmod = os.getenv('LMOD_CMD')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to implement in this PR, but we should add some error handling here.
What if $LMOD_CMD is not set (or points to a non-existing path)?
What if the exit code of the Lmod command is non-zero?

scripts/module_list/module_overview.py Outdated Show resolved Hide resolved
"""
return data[~np.char.endswith(data, ":") &
~np.char.startswith(data, "env/") &
~np.char.startswith(data, "cluster/default")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good enough for now, but a better way to filter these would be to:

  • for each module, keep track of which location it was found in, so iterate over output of module avail line by line, and build up a dict with path as key, list of modules as value;
  • filter based on /etc/modulefiles/vsc (which covers env/* and cluster/*) to discriminate between software module and cluster modules;

scripts/module_list/module_overview.py Outdated Show resolved Hide resolved
scripts/module_list/module_overview.py Outdated Show resolved Hide resolved
scripts/module_list/module_overview.py Outdated Show resolved Hide resolved
.github/workflows/script_module_list.yml Outdated Show resolved Hide resolved
.github/workflows/script_module_list.yml Outdated Show resolved Hide resolved
scripts/module_list/README.md Outdated Show resolved Hide resolved
scripts/README.md Outdated Show resolved Hide resolved
scripts/module_list/README.md Outdated Show resolved Hide resolved
scripts/module_list/tests/test_md.py Outdated Show resolved Hide resolved
scripts/module_list/tests/data/lmod_mock.sh Outdated Show resolved Hide resolved
scripts/module_list/tests/data/lmod_mock.sh Outdated Show resolved Hide resolved
scripts/module_list/tests/data/lmod_mock.sh Outdated Show resolved Hide resolved
scripts/module_list/tests/data/lmod_mock.sh Outdated Show resolved Hide resolved
@milachae milachae requested a review from boegel August 14, 2023 09:10
@milachae
Copy link
Contributor Author

@boegel
I resolved all the suggestions

scripts/module_overview/tests/test_md.py Outdated Show resolved Hide resolved
.github/workflows/script_module_list.yml Outdated Show resolved Hide resolved
scripts/module_overview/README.md Outdated Show resolved Hide resolved
scripts/module_overview/README.md Outdated Show resolved Hide resolved
scripts/module_overview/README.md Outdated Show resolved Hide resolved
scripts/module_overview/README.md Outdated Show resolved Hide resolved
scripts/module_overview/README.md Outdated Show resolved Hide resolved
scripts/module_overview/README.md Show resolved Hide resolved
md_file = MdUtils(file_name='module_overview.md', title='Overview of available modules per cluster')
data = modules_ugent()
generate_module_table(data, md_file)
md_file.create_md_file()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also print path to file that was created?

Co-authored-by: Kenneth Hoste <kenneth.hoste@ugent.be>
@boegel boegel marked this pull request as ready for review August 14, 2023 14:31
@boegel boegel merged commit 55c8b55 into hpcugent:main Aug 14, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants