Add lit-llama benchmarks (logits, autoregressive generation, lora fine tuning) #1730
Conversation
…e tuning) Signed-off-by: Edward Z. Yang <ezyang@meta.com>
| def train(self): | ||
| logits = self.model(*self.example_inputs) | ||
| logits.sum().backward() | ||
| # meh this sucks |
There was a problem hiding this comment.
xd, this might be a good dataset https://huggingface.co/datasets/OpenAssistant/oasst1
Even finetuning on two examples of questions you make up might be not bad as a sanity
There was a problem hiding this comment.
Will fix this later, I think. Not needed for dynamo benchmarks.
| def eval(self): | ||
| self.model.eval() | ||
| with torch.no_grad(): | ||
| y = self.model(*self.example_inputs) |
There was a problem hiding this comment.
do you mind printing the input prompt and the output, will be nice to do vibe checks later
There was a problem hiding this comment.
Hmm, but I don't want to print it here, because then the detokenization would also count as part of the benchmark?
| self.model = GenerationWrapper(self.model) | ||
| tokenizer = Tokenizer(os.path.join(LIT_LLAMA_PATH, "checkpoints/lit-llama/tokenizer.model")) | ||
| # max_new_tokens matches lit-llama/generate.py | ||
| self.example_inputs = (tokenizer.encode("The meaning of life is", bos=True, eos=False, device=device), 50) |
There was a problem hiding this comment.
is 50 the max number of tokens to generate?
|
@xuzhao9 this will be a common workflow for LLM work (SAM is similar today), it might make sense to cache these files in a github artifact or an S3 bucket if github has data size limits |
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
|
Do we have any precedent for hosting it in S3? I am happy to set it up if there is some example of doing it. |
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
I am not sure if it requires legal review for that. |
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
|
So, it seems like we can only run this benchmark on the A100s anyway, so I'm going to disable the A10G configuratoin |
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
| class Model(BenchmarkModel): | ||
| task = NLP.LANGUAGE_MODELING | ||
| DEFAULT_EVAL_BSIZE = 1 | ||
| DEFAULT_TRAIN_BSIZE = 32 |
There was a problem hiding this comment.
Curious why the default train batch size is 32?
There was a problem hiding this comment.
I think I should just delete this, it's meaningless, you can't train 7B without some sort of distribution haha
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
|
@ezyang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
oh thank god, pr-test is finally passing |
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
|
@ezyang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Signed-off-by: Edward Z. Yang ezyang@meta.com