torchbench is now a library

[msaroufim]

import torchbenchmark.models.densenet121
model, example_inputs = torchbenchmark.models.densenet121.Model(test="eval", device="cuda", batch_size=1).get_module()
model(*example_inputs)

So making the above example work with import torchbenchmark alone is a bit trickier there's a lot of relative imports that need to be fixed in pretty much every single file

I also added a simple test_imports.py file to make sure this doesn't break anything, I can setup a standalone github action or plug into an existing one (would rather do the latter to avoid another job that needs to run install.py again

This also solves @voznesenskym favorite design pattern from https://github.com/pytorch/pytorch/blob/main/benchmarks/dynamo/torchbench.py

    for torchbench_dir in (
        "./torchbenchmark",
        "../torchbenchmark",
        "../torchbench",
        "../benchmark",
        "../../torchbenchmark",
        "../../torchbench",
        "../../benchmark",
    ):

[xuzhao9]

Looking good! Though I am not sure what we should do about install.py. Just running pip install git+https://www.github.com:pytorch/benchmark.git does not mean we can run all the models. We still need to run install.py to install all the model dependencies, download data to the directory, etc.

[msaroufim]

I think you should keep install.py as is since you do expect users to bring their own torch versions and there's a large number of dependencies that will be included in the torchbench package

So user should do

# Install torch/audio/text from source or nightlies
python install.py
pip install .

[xuzhao9]

Score is only for archiving purpose and is to be removed - we can skip importing it.

[facebook-github-bot]

@msaroufim has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[xuzhao9]

I think you should keep install.py as is since you do expect users to bring their own torch versions and there's a large number of dependencies that will be included in the torchbench package

So user should do

# Install torch/audio/text from source or nightlies
python install.py
pip install .

Sounds reasonable, it is better to clarify in the README.md that they need to run both python install.py and pip install .

[facebook-github-bot]

@msaroufim has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@msaroufim has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@msaroufim merged this pull request in f2d615b.