Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Performance] A@B.T - The GEMM performance with the column major B matrix is not as good as row major B matrix. #2354

Open
chengjunlu opened this issue Sep 26, 2024 · 5 comments · May be fixed by #2834
Open

[Performance] A@B.T - The GEMM performance with the column major B matrix is not as good as row major B matrix. #2354

chengjunlu opened this issue Sep 26, 2024 · 5 comments · May be fixed by #2834
Assignees
@alexbaden
Labels
enhancement New feature or request performance umbrella
Milestone
4.0 [Performance]...

Comments

@chengjunlu
Copy link
Contributor

The performance gap is found in #2347

Need to investigate root cause of the performance drops of the column major B matrix case.
Roughly 1.5x worse than the row major B matrix case.

(I): Detected 7680 spills, recompiling the kernel using large GRF mode
(I): Kernel has now 0 spills
✅ Triton and Torch match
Time for torch: 0.31633758544921875 ms
Time for triton: 0.44517597556114197 ms
Compute A x B.T
OpenCL API not available for this operation
OpenCL API not available for this operation
OpenCL API not available for this operation
OpenCL API not available for this operation
(I): Detected 7680 spills, recompiling the kernel using large GRF mode
(I): Kernel has now 0 spills
✅ Triton and Torch match
Time for torch: 0.3375360071659088 ms
Time for triton: 0.6348815560340881 ms
@chengjunlu chengjunlu mentioned this issue Sep 26, 2024
Merged
@vlad-penkin vlad-penkin added performance enhancement New feature or request labels Sep 27, 2024
@vlad-penkin vlad-penkin added this to the 4.0 [Performance] Core milestone Sep 27, 2024
@Egor-Krivov Egor-Krivov mentioned this issue Oct 4, 2024
Closed
@Egor-Krivov
Copy link
Contributor

I think this issue is essential for GEMM perf. Very often weights are stored with K dimensions as the last. Even pytorch linear layer does that: weight torch.Tensor – the learnable weights of the module of shape : (out_features, in_features)

https://pytorch.org/docs/stable/generated/torch.nn.Linear.html

@alexbaden
Copy link
Contributor

Adding to this, if the A matrix is column-major we have similar problems.

@alexbaden alexbaden mentioned this issue Oct 11, 2024
Closed
@Egor-Krivov
Copy link
Contributor

Egor-Krivov commented Oct 11, 2024 •
edited
Loading

We now have microbenchmarks to track this performance. Currently GeoMean for onednn is ~90-100TFLOPs for both cases of A.T@B and for A@B.T.

A@B.T for triton currently stands at ~60TFLOPs. Dashboard gemm-bt
A.T@B for triton currently stands at ~30TFLOPs, it significantly improved and was ~15TFLOPs recently. Dashboard gemm-at

So onednn is 1.5 times faster for B.T and 3 times faster for A.T

@Egor-Krivov
Copy link
Contributor

@alexbaden Should we change the title to reflect issue with A.T as well or create separate issue for that case?

@vlad-penkin vlad-penkin changed the title [Performance] The GEMM performance with the column major B matrix is not as good as row major B matrix. [Performance] A@B.T - The GEMM performance with the column major B matrix is not as good as row major B matrix. Oct 21, 2024
@alexbaden
Copy link
Contributor

Current Triton tiling strategy for DPAS for AxBT:

image

oneDNN tiling strategy mapped to Triton (thanks to @Jianhui-Li and @chengjunlu ) :

image

I plan to try to implement the oneDNN strategy in Triton.

@alexbaden alexbaden linked a pull request Nov 26, 2024 that will close this issue
Use order from A matrix when determining DPAS layout #2834
Draft
@alexbaden alexbaden mentioned this issue Dec 6, 2024
Merged
alexbaden added a commit that referenced this issue Dec 9, 2024
@alexbaden
Permute the pass pipeline to coalesce before setting up the matmul (#…
cacfe10 
…2956)

Required for #2834 

Two reasons to do this - one, it properly tags the layouts with their
memory order very early in the TTGIR pipeline. And two, it moves our
TTGIR pipeline closer to upstream. I am splitting the change to isolate
any regressions or undesired behavior caused by this change vs changing
the DPAS layouts in #2834.

cc #2354
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexbaden alexbaden

Labels
enhancement New feature or request performance umbrella
Projects
None yet
Milestone
4.0 [Performance] Core
Development

Successfully merging a pull request may close this issue.

Use order from A matrix when determining DPAS layout intel/intel-xpu-backend-for-triton
4 participants
@Egor-Krivov @alexbaden @chengjunlu @vlad-penkin