Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add end-to-end automated test workflow #63

Closed
russellb opened this issue Jun 24, 2024 · 1 comment · Fixed by #69 or #174
Closed

Add end-to-end automated test workflow #63

russellb opened this issue Jun 24, 2024 · 1 comment · Fixed by #69 or #174
Labels
CI/CD Affects CI/CD configuration testing Relates to testing

Comments

@russellb
Copy link
Member

Once this library is being consumed by instructlab/instructlab, it would be helpful to automatically test changes to this library in an end-to-end run of the ilab workflow. You can copy this workflow that I added to the sdg repo to do this. It should work with no modifications, though it does say "instructlab-sdg" in one comment.

instructlab/sdg#33
russellb added a commit to russellb/instructlab-training that referenced this issue Jun 25, 2024
@russellb
github: Add e2e test workflow
72de55e 
This workflow runs an end-to-end test using `ilab` on a github rubber
with a single NVIDIA GPU. It will install `instructlab` from `main`
plus this library with whatever proposed changes are included in a
given PR.

Closes instructlab#63

Signed-off-by: Russell Bryant <rbryant@redhat.com>
@russellb russellb mentioned this issue Jun 25, 2024
Merged
@RobotSail RobotSail mentioned this issue Jun 25, 2024
Closed
@russellb russellb reopened this Jun 26, 2024
@russellb
Copy link
Member Author

This was reverted in #76. My comment there explains why this is relevant:

Testing the user end-to-end instructlab workflow is the point of this workflow. It will give you an early indication if your changes to the library break the CLI. Sometimes that might be on purpose, but in case it's not, it will let you know before you publish a release.

This is not intended to replace other functional testing aimed at the library more directly, but the full end-to-end workflow is important.

I'm OK if you want to disable it for the moment, but it needs to go back on at some point. A better way to do that is to just edit the workflow to turn it off instead of removing it from the repo.

russellb added a commit to russellb/instructlab-training that referenced this issue Jun 26, 2024
@russellb
Revert "remove incorrect end-to-end implementation"
f9bfcee 
This reverts commit ebc5e31 from instructlab#76.

Here is my comment on instructlab#76 that better explains how this workflow fits
in to the bigger picture:

> Testing the user end-to-end instructlab workflow is the point of
> this workflow. It will give you an early indication if your changes to
> the library break the CLI. Sometimes that might be on purpose, but in
> case it's not, it will let you know before you publish a release.
>
> This is not intended to replace other functional testing aimed at the
> library more directly, but the full end-to-end workflow is important.
>
> I'm OK if you want to disable it for the moment, but it needs to go
> back on at some point. A better way to do that is to just edit the
> workflow to turn it off instead of removing it from the repo.

Issue instructlab#63

Signed-off-by: Russell Bryant <rbryant@redhat.com>
@russellb russellb mentioned this issue Jun 26, 2024
Closed
russellb added a commit to russellb/instructlab-training that referenced this issue Jun 27, 2024
@russellb
Revert "remove incorrect end-to-end implementation"
5f4d371 
This reverts commit ebc5e31 from instructlab#76.

Here is my comment on instructlab#76 that better explains how this workflow fits
in to the bigger picture:

> Testing the user end-to-end instructlab workflow is the point of
> this workflow. It will give you an early indication if your changes to
> the library break the CLI. Sometimes that might be on purpose, but in
> case it's not, it will let you know before you publish a release.
>
> This is not intended to replace other functional testing aimed at the
> library more directly, but the full end-to-end workflow is important.
>
> I'm OK if you want to disable it for the moment, but it needs to go
> back on at some point. A better way to do that is to just edit the
> workflow to turn it off instead of removing it from the repo.

The previous commit also removed spellcheck (though it wasn't
mentioned). I have left it out in this PR.

Issue instructlab#63

Signed-off-by: Russell Bryant <rbryant@redhat.com>
nathan-weinberg pushed a commit to nathan-weinberg/training that referenced this issue Jun 27, 2024
@russellb @nathan-weinberg
github: Add e2e test workflow
b903388 
This workflow runs an end-to-end test using `ilab` on a github rubber
with a single NVIDIA GPU. It will install `instructlab` from `main`
plus this library with whatever proposed changes are included in a
given PR.

Closes instructlab#63

Signed-off-by: Russell Bryant <rbryant@redhat.com>
nathan-weinberg pushed a commit to nathan-weinberg/training that referenced this issue Jun 27, 2024
@russellb @nathan-weinberg
github: Add e2e test workflow
e73fcfe 
This workflow runs an end-to-end test using `ilab` on a github rubber
with a single NVIDIA GPU. It will install `instructlab` from `main`
plus this library with whatever proposed changes are included in a
given PR.

Closes instructlab#63

Signed-off-by: Russell Bryant <rbryant@redhat.com>
russellb added a commit to russellb/instructlab-training that referenced this issue Jun 28, 2024
@russellb
Revert "remove incorrect end-to-end implementation"
eb71ef7 
This reverts commit ebc5e31 from instructlab#76.

Here is my comment on instructlab#76 that better explains how this workflow fits
in to the bigger picture:

> Testing the user end-to-end instructlab workflow is the point of
> this workflow. It will give you an early indication if your changes to
> the library break the CLI. Sometimes that might be on purpose, but in
> case it's not, it will let you know before you publish a release.
>
> This is not intended to replace other functional testing aimed at the
> library more directly, but the full end-to-end workflow is important.
>
> I'm OK if you want to disable it for the moment, but it needs to go
> back on at some point. A better way to do that is to just edit the
> workflow to turn it off instead of removing it from the repo.

The previous commit also removed spellcheck (though it wasn't
mentioned). I have left it out in this PR.

Issue instructlab#63

Signed-off-by: Russell Bryant <rbryant@redhat.com>
@nathan-weinberg nathan-weinberg linked a pull request Aug 20, 2024 that will close this issue
feat: add e2e test for instructlab CI #174
Merged
@nathan-weinberg nathan-weinberg added CI/CD Affects CI/CD configuration testing Relates to testing labels Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CI/CD Affects CI/CD configuration testing Relates to testing
Projects
None yet
Milestone
No milestone
2 participants
@russellb @nathan-weinberg