Add Job dispatcher #360
Merged
Add Job dispatcher #360
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yoshipon
requested changes
Mar 25, 2024
|
今のコード拝見してて思ったんですが，ロジックはExecutorに全部うつして，AbciJobはデータ (dataclass) のみにした方が見通しが良いですかね…？ |
yoshipon
requested changes
Apr 30, 2024
aramoto99
force-pushed
the
feature/v2-jobexecutor
branch
from
e162b5a to
7a059f7
Compare
yoshipon
requested changes
May 19, 2024
|
@ yoshipon |
yoshipon
added a commit
that referenced
this pull request
Aug 26, 2024
* Add nelder-mead sampler (#346) * add nelder_mead_sampler.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add -e. (#371) * Support for suggest_int in neldermead sampler. (#373) * Support for suggest_int. * Fix for ruff. * Add Job dispatcher (#360) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * fix test * fix test * fix test * remove __future__.annotations * A bit simplify qstat_xml * Simplify tests --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add ignore_missing_imports. (#376) * Draft Implementation of CLI Command (#369) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * Add 'start.py' * Fix code format * update pyproject.toml * Remove unnecessary items from config * Add random seed option * Refactor examples/start.py into aiaccel/apps/optimize.py * fix lint errors and warnings * remove apps/config * fix typo * change config.yaml format * fix apps/optimizer * fix HparamManager __init__ * fix HparamsManager * Refactor Suggest class to use generics for improved type safety * auto fix by ruff * add optuna suggest wrapper * format fix * Update wrapper.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add nm docstring and move file (#381) * Add docstring. * Move nelder_mead_sampler.py. * Fix path. * Move test files. * autoformat and fix mypy error * Add v2 torch docstring (#384) * add docstring * Update abci_environment.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> --------- Co-authored-by: KanaiYuma-aist <105629713+KanaiYuma-aist@users.noreply.github.com> Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com>
aramoto99
added a commit
that referenced
this pull request
Sep 2, 2024
* Add nelder-mead sampler (#346) * add nelder_mead_sampler.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add -e. (#371) * Support for suggest_int in neldermead sampler. (#373) * Support for suggest_int. * Fix for ruff. * Add Job dispatcher (#360) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * fix test * fix test * fix test * remove __future__.annotations * A bit simplify qstat_xml * Simplify tests --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add ignore_missing_imports. (#376) * Draft Implementation of CLI Command (#369) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * Add 'start.py' * Fix code format * update pyproject.toml * Remove unnecessary items from config * Add random seed option * Refactor examples/start.py into aiaccel/apps/optimize.py * fix lint errors and warnings * remove apps/config * fix typo * change config.yaml format * fix apps/optimizer * fix HparamManager __init__ * fix HparamsManager * Refactor Suggest class to use generics for improved type safety * auto fix by ruff * add optuna suggest wrapper * format fix * Update wrapper.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add nm docstring and move file (#381) * Add docstring. * Move nelder_mead_sampler.py. * Fix path. * Move test files. * autoformat and fix mypy error * Add v2 torch docstring (#384) * add docstring * Update abci_environment.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> --------- Co-authored-by: KanaiYuma-aist <105629713+KanaiYuma-aist@users.noreply.github.com> Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com>
aramoto99
added a commit
that referenced
this pull request
Sep 2, 2024
* Add nelder-mead sampler (#346) * add nelder_mead_sampler.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add -e. (#371) * Support for suggest_int in neldermead sampler. (#373) * Support for suggest_int. * Fix for ruff. * Add Job dispatcher (#360) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * fix test * fix test * fix test * remove __future__.annotations * A bit simplify qstat_xml * Simplify tests --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add ignore_missing_imports. (#376) * Draft Implementation of CLI Command (#369) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * Add 'start.py' * Fix code format * update pyproject.toml * Remove unnecessary items from config * Add random seed option * Refactor examples/start.py into aiaccel/apps/optimize.py * fix lint errors and warnings * remove apps/config * fix typo * change config.yaml format * fix apps/optimizer * fix HparamManager __init__ * fix HparamsManager * Refactor Suggest class to use generics for improved type safety * auto fix by ruff * add optuna suggest wrapper * format fix * Update wrapper.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add nm docstring and move file (#381) * Add docstring. * Move nelder_mead_sampler.py. * Fix path. * Move test files. * autoformat and fix mypy error * Add v2 torch docstring (#384) * add docstring * Update abci_environment.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> --------- Co-authored-by: KanaiYuma-aist <105629713+KanaiYuma-aist@users.noreply.github.com> Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com>
yoshipon
added a commit
that referenced
this pull request
Sep 20, 2024
* Add torch goodies * Add type annotations * Add small things * update for mypy * Add several tests * add test for hdf5_dataset.py * Update workflows * Add tests * update workflows * add dependencies * update train.py * add opt_lightning_module * add opt_lightning_module * Update error messages! * Update hdf5_dataset to take grp_list as argument * sort grp_list * update v2 torch (#383) * Add nelder-mead sampler (#346) * add nelder_mead_sampler.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add -e. (#371) * Support for suggest_int in neldermead sampler. (#373) * Support for suggest_int. * Fix for ruff. * Add Job dispatcher (#360) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * fix test * fix test * fix test * remove __future__.annotations * A bit simplify qstat_xml * Simplify tests --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add ignore_missing_imports. (#376) * Draft Implementation of CLI Command (#369) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * Add 'start.py' * Fix code format * update pyproject.toml * Remove unnecessary items from config * Add random seed option * Refactor examples/start.py into aiaccel/apps/optimize.py * fix lint errors and warnings * remove apps/config * fix typo * change config.yaml format * fix apps/optimizer * fix HparamManager __init__ * fix HparamsManager * Refactor Suggest class to use generics for improved type safety * auto fix by ruff * add optuna suggest wrapper * format fix * Update wrapper.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add nm docstring and move file (#381) * Add docstring. * Move nelder_mead_sampler.py. * Fix path. * Move test files. * autoformat and fix mypy error * Add v2 torch docstring (#384) * add docstring * Update abci_environment.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> --------- Co-authored-by: KanaiYuma-aist <105629713+KanaiYuma-aist@users.noreply.github.com> Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add torch goodies * Add type annotations * Add small things * update for mypy * Add several tests * add test for hdf5_dataset.py * Update workflows * Add tests * update workflows * add dependencies * update train.py * add opt_lightning_module * Update error messages! * add opt_lightning_module * Update hdf5_dataset to take grp_list as argument * sort grp_list * update v2 torch (#383) * Add nelder-mead sampler (#346) * add nelder_mead_sampler.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add -e. (#371) * Support for suggest_int in neldermead sampler. (#373) * Support for suggest_int. * Fix for ruff. * Add Job dispatcher (#360) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * fix test * fix test * fix test * remove __future__.annotations * A bit simplify qstat_xml * Simplify tests --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add ignore_missing_imports. (#376) * Draft Implementation of CLI Command (#369) * Update aiaccel/job * fix lint error * fix aiaccel/job/dispatcher.py * wip * Fix issues pointed out in review * rename * fix test * fix for test * fix for test * fix for test * fix for test * Reflected code review feedback * fix test * fix test * refactoring * Run code formatter ruff * refactoring * Run code formatter ruff * fix lint.yaml * fix pypoject.toml * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * fix mypy.ini * Refactor abci jobs * Update pyproject.toml * adjusted to work in the ABCI environment. * fix lint error * fix test * fix test * Changed from unittest to pytest * Incorporate revisions based on review feedback * Add test items * Add 'start.py' * Fix code format * update pyproject.toml * Remove unnecessary items from config * Add random seed option * Refactor examples/start.py into aiaccel/apps/optimize.py * fix lint errors and warnings * remove apps/config * fix typo * change config.yaml format * fix apps/optimizer * fix HparamManager __init__ * fix HparamsManager * Refactor Suggest class to use generics for improved type safety * auto fix by ruff * add optuna suggest wrapper * format fix * Update wrapper.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * Add nm docstring and move file (#381) * Add docstring. * Move nelder_mead_sampler.py. * Fix path. * Move test files. * autoformat and fix mypy error * Add v2 torch docstring (#384) * add docstring * Update abci_environment.py --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> --------- Co-authored-by: KanaiYuma-aist <105629713+KanaiYuma-aist@users.noreply.github.com> Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> * fix confrict * Fixed consistency in revision history * Fixed consistency in revision history(2) --------- Co-authored-by: Yoshiaki Bando <yoshipon@users.noreply.github.com> Co-authored-by: Yoshiaki Bando <3883362-yoshipon@users.noreply.gitlab.com> Co-authored-by: KanaiYuma-aist <105629713+KanaiYuma-aist@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
aiaccel/job/*に，ABCI向けのジョブディスパッチャを追加.