ROCm support #295
ROCm support #295
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| @@ -0,0 +1,34 @@ | |||
| ## Testing | |||
There was a problem hiding this comment.
I think this is equivalent to the integration snapshoting logic in router/tests?
| model = AutoModel.from_pretrained(model_path).to(dtype).to(device) | ||
| self.hidden_size = model.config.hidden_size | ||
| self.pooling_mode = pooling_mode |
There was a problem hiding this comment.
Can you add mean pooling / CLS pooling forking L46?
There was a problem hiding this comment.
CLS pooling was already there, I added mean pooling.
What is L46?
There was a problem hiding this comment.
@OlivierDehaene, I created a pr to fix this
…-inference into rocm-support
|
any chance this MR is merged soon? |
|
@almersawi it is in good shape to me. |
|
Hey guys! thanks for this MR. We have been waiting for it for some time now. Any idea when this MR will be merged ? |
|
Can please anyone provide build steps for docker image with this branch. /opt/conda/lib/python3.10/site-packages/torch/include/ATen/hip/HIPContextLight.h:20:10: fatal error: 'hipsolver/hipsolver.h' file not found Command I executed : sudo docker build -f Dockerfile-rocm -t nims123/tei_amd . |
|
Applied minor fixes to successfully build the Docker image. PR #403 |
|
any ETA on this merge? |
As per title. Support AMD GPUs through TEI backend.
For now, only embedding model with cls/mean pooling is tested.
MI210/MI250/MI300 can dispatch on CK flash attention 2, but other GPUs will default to manual attention implem (or SDPA). Only bert looks to be supported in the python backend.