Update ROCM libs and improvements #2358
Conversation
|
Hi @mht-sharma 👋 Just checking in on this: are you still working on it or is this something we should consider closed? And my intention is by no means to say that we're in a hurry 👍 |
Hi @ErikKaum , yes I am currently working on this, with a few improvements and fixes still pending. I am working with AMD to ensure these updates are finalized soon. |
mht-sharma
changed the title
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
mht-sharma
changed the title
| torch.empty( | ||
| torch.zeros( | ||
| (num_blocks, num_heads, head_size // x, BLOCK_SIZE, x), | ||
| dtype=dtype, | ||
| device=device, | ||
| ), | ||
| torch.empty( | ||
| torch.zeros( | ||
| (num_blocks, num_heads, head_size, BLOCK_SIZE), | ||
| dtype=dtype, | ||
| device=device, |
There was a problem hiding this comment.
This change is required for custom PA kernel in ROCM.
|
@OlivierDehaene @Narsil could you please review the PR and merge |
server/text_generation_server/models/custom_modeling/flash_cohere_modeling.py
Outdated
Show resolved
Hide resolved
|
Closing this in favour of #2579 |
What does this PR do?
This PR introduces various library updates to address breaking changes, including optimisations for ROCm and custom kernels for low-batch-size GEMM and Paged attention. Key improvements are as follows:
rocm/vllmcommit.