Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducibility: Features for Artifact Evaluation #52

Open
4 tasks
nicolas-kuechler opened this issue Jul 7, 2022 · 0 comments
Open
4 tasks

Reproducibility: Features for Artifact Evaluation #52

nicolas-kuechler opened this issue Jul 7, 2022 · 0 comments

Comments

@nicolas-kuechler
Copy link
Owner

nicolas-kuechler commented Jul 7, 2022

For artifact evaluations, we should provide more features for reproducibility:

  • run a suite based on suite_design in a results folder rather than on the design
  • store the commit hash or something like this next to the results (how to deal with local changes? maybe have a "reproducible" flag that enforces that you don't have changes?)
  • provide a way to run a meaningful subset of the results?
  • store info on which cloud the experiment was executed (introduce check that when calling last, that the cloud still matches)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant