Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BE] consolidate 4-GPU integration tests into 8-GPU tests and reduce frequency #745

Closed
wants to merge 3 commits into from
Closed

[BE] consolidate 4-GPU integration tests into 8-GPU tests and reduce frequency #745

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
71b96ed
consolidate 4-GPU integration tests into 8-GPU tests and reduce frequ…
tianyu-l Dec 17, 2024
da432eb
Update on "[BE] consolidate 4-GPU integration tests into 8-GPU tests …
tianyu-l Dec 17, 2024
5a686d5
Update on "[BE] consolidate 4-GPU integration tests into 8-GPU tests …
tianyu-l Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 0 additions & 46 deletions .github/workflows/integration_test_4gpu.yaml
View file
Open in desktop

This file was deleted.

10 changes: 7 additions & 3 deletions .github/workflows/integration_test_8gpu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ on:
branches: [ main ]
pull_request:
schedule:
# Runs nightly
- cron: '0 0 * * *'
# Runs every 6 hours
- cron: '0 */6 * * *'
concurrency:
group: unit-test${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_number || github.ref }}
cancel-in-progress: true
Expand All @@ -21,7 +21,7 @@ jobs:
with:
runner: linux.g5.48xlarge.nvidia.gpu
gpu-arch-type: cuda
gpu-arch-version: "12.1"
gpu-arch-version: "12.4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has someone been running local tests with 12.4 CUDA? Does it work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't. To be honest I'm not sure.
This change is mainly for consistency with https://github.com/pytorch/torchtitan/pull/745/files#diff-e327f3f247423713ee949ef4eef6b82de392abca8c53137159d82f073510c4f9R39

# This image is faster to clone than the default, but it lacks CC needed by triton
# (1m25s vs 2m37s).
docker-image: torchtitan-ubuntu-20.04-clang12
Expand All @@ -37,5 +37,9 @@ jobs:
pip config --user set global.progress_bar off

python -m pip install --force-reinstall --pre torch --index-url https://download.pytorch.org/whl/nightly/cu124

# install torchtitan to test the files in ./scripts
python -m pip install -e .

mkdir artifacts-to-be-uploaded
python ./tests/integration_tests.py artifacts-to-be-uploaded --ngpu 8
3 changes: 1 addition & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
[![4 GPU Integration Test](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_4gpu.yaml/badge.svg?branch=main)](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_4gpu.yaml?query=branch%3Amain)
[![8 GPU Integration Test](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu.yaml/badge.svg?branch=main)](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu.yaml?query=branch%3Amain)
[![Integration Tests](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu.yaml/badge.svg?branch=main)](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu.yaml?query=branch%3Amain)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will we only use the 8gpu AWS instance for CI from now on?

Copy link
Contributor Author

@tianyu-l tianyu-l Dec 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the idea of this PR. Please share concerns you may have!

# torchtitan

Expand Down
6 changes: 1 addition & 5 deletions tests/integration_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,10 +472,6 @@ def run_tests(args):
f"Skipping test {test_flavor.test_name} that requires {test_flavor.ngpu} gpus,"
f" because --ngpu arg is {args.ngpu}"
)
elif args.ngpu == 8 and test_flavor.ngpu != 8:
logger.info(
f"Skipping non-8gpu test {test_flavor.test_name} on 8-gpu runner"
)
else:
run_test(test_flavor, full_path, args.output_dir)

Expand All @@ -489,7 +485,7 @@ def main():
default="all",
help="test to run, acceptable values: `test_name` in `build_test_list` (default: all)",
)
parser.add_argument("--ngpu", default=4, type=int)
parser.add_argument("--ngpu", default=8, type=int)
args = parser.parse_args()

if not os.path.exists(args.output_dir):
Expand Down
Loading