Cuda heat example w quaditer #913

Abdelrahman912 · 2024-05-14T12:06:22Z

Heat Example Prototype using CUDA.jl and StaticCellValues

koehlerson · 2024-05-15T08:51:36Z

docs/src/literate-tutorials/gpu_qp_heat_equation.jl

+end
+
+Kgpu = CUDA.zeros(dh.ndofs.x,dh.ndofs.x)
+gpu_dh = GPUDofHandler(dh)


Maybe that's the plan anyway but wouldn't it be nicer to write Adapt rules for DofHandler and Grid which return adapt_structure for the GPU structs?

function Adapt.adapt_structure(to,dh:DofHandler) return adapt_structure(GPUDofHandler(cu(Int32.(dh.cell_dofs)),GPUGrid(dh.grid))) end

and then just use the "normal" structs from a user perspective that are automatically converted when needed

I am still a bit split about this, because it contains a performance pitfalls. If we have repeated assembly, then the dof handler will be converted and copied to GPU for each assembly instead of only once.

Am I missing something or wouldn't the adapt call not also happen for each assembly kernel launch for the GPUDofHandler in that case?

This should not happen, because we do not need to adapt the GPUDofHandler (it is already a GPU datastructure).

KristofferC · 2024-05-15T10:01:13Z

docs/src/literate-tutorials/gpu_qp_heat_equation.jl

+end
+
+
+gm = static_cellvalues.gm


Maybe this is still a work in progress but the mix of functions and global variables make it kind of confusing to read the code.

Abdelrahman912 · 2024-05-23T14:18:14Z

What I did for now, and it's still work in progress:

I added some higher-level abstractions and some restructuring to match the original example (still need some refactoring).
I used the QuadratureValuesIterator and edited the StaticCellValue object to be compatible with the GPU.

This still work in progress and as my discussion with @termi-official last week I still need to work on the assembler, coloring algorthim.

Some problems I have encountered that might be so straightforward to tackle:

Grid object contains Dict type which is not GPU compatible.

termi-official · 2024-05-23T19:29:05Z

Great to see some quick progress here!

Some problems I have encountered that might be so straightforward to tackle:
1. `Grid` object contains `Dict` type which is not GPU compatible.

I think that is straight forward to solve. We never really need the Dicts directly during assembly. We should be able to get away by just convert the Vectors (once) to GPUVectors and run the assembly with these. This might require 2 structs. One holding the full information (e.g. GPUGrid) and one which we use in the kernels (e.g. GPUGridView). Maybe the latter could be something like

struct GPUGridView{TEA, TNA, TSA <: Union{Nothing, <:AbstractVector{Int}, <: AbstractVector{FaceIndex}, ..., TCA} <: AbstractGrid (?)
    cells::TEA
    nodes::TNA
    subdomain::TSA
    color::TCA
end

where subdomain just holds the data which we want to iterate over (or nothing for all cells) and color is a vector for elements with one color of the current subdomain.

KnutAM · 2024-05-23T19:44:24Z

A longer-term thing just to throw out the idea, but perhaps a more slim Grid could be nice?

struct Grid{dim, C, T, CV, NV, S}
    cells::CV
    nodes::NV
    gridsets::S
    function Grid(cells::AbstractVector{C}, nodes::AbstractVector{Node{dim, T}}, gridsets) where {C, dim, T}
        return new{dim, C, T, typeof(cells), typeof(nodes), typeof(sets)}(cells, nodes, gridsets)
     end
end
struct GridSets
    facetsets::Dict{String, OrderedSet{FacetIndex}}
    cellsets::Dict{String, OrderedSets{Int}}
    ....
end

allowing also gridsets=nothing

termi-official · 2024-06-04T14:18:29Z

A longer-term thing just to throw out the idea, but perhaps a more slim Grid could be nice?

struct Grid{dim, C, T, CV, NV, S}
    cells::CV
    nodes::NV
    gridsets::S
    function Grid(cells::AbstractVector{C}, nodes::AbstractVector{Node{dim, T}}, gridsets) where {C, dim, T}
        return new{dim, C, T, typeof(cells), typeof(nodes), typeof(sets)}(cells, nodes, gridsets)
     end
end
struct GridSets
    facetsets::Dict{String, OrderedSet{FacetIndex}}
    cellsets::Dict{String, OrderedSets{Int}}
    ....
end

allowing also gridsets=nothing

I thought of this quite a bit already, whether we should have our grid in the form

struct Grid{dim, C, T, CV, NV, S, TT}
    cells::CV
    nodes::NV
    subdomain_info::S
    function Grid(cells::AbstractVector{C}, nodes::AbstractVector{Node{dim, T}}, subdomain_info) where {C, dim, T}
        return new{dim, C, T, typeof(cells), typeof(nodes), typeof(sets)}(cells, nodes, subdomain_info)
     end
end

where subdomain info contains any kind of subdomain information. This could also include potentially some optional topology information which we need for some problems. In the simplest case it would be just facesets and cellsets.

However, we should do this in a separate PR. What do you think @fredrikekre ?

src/Grid/grid_generators.jl

Abdelrahman912 · 2024-06-20T22:22:53Z

So far:

example implementation using the coloring algorithm and I did my best to follow the same abstraction as in the CPU case (also one could circumvent this by introducing metaprogramming to set up the kernel before launching but this might be relevant for later discussion)
I had to implement a custom assembler (naive implementation) gpu_assembler because the already existing one cannot be used because the permutation and sorteddofs attributes are mutable (regarding to the elements and the size) so it's anly valid for sequential code, not to mention the resize!
Also, setting an index for CuSparseMatrixCSC is not allowed inside a kernel (ref: https://discourse.julialang.org/t/cuda-jl-atomic-addition-error-to-a-sparse-array-inside-cuda-kernel/78789/2), so I write a very naive GPUSparseMatrixCSC.
Final observation, in create_sparsity_pattern it only create sparse matrix with nzvals of type Float64 as follows :

    - K = spzeros!!(Float64, I, J, ndofs(dh), ndofs(dh)) #old code
    + K = spzeros!!(T, I, J, ndofs(dh), ndofs(dh)) # my proposal

I don't know whether this was intended or what but I found it worth mentioning.

termi-official · 2024-06-20T22:33:12Z

Thanks for putting this so far together! Some quick comments before the next meeting for you.

2. I had to implement a custom assembler (naive implementation) `gpu_assembler` because the already existing one cannot be used because the `permutation` and `sorteddofs` attributes are mutable (regarding to the elements and the size) so it's anly valid for sequential code, not to mention the `resize!`

Indeed and I started refactoring some of the assembly code here #916 . I also think that we cannot get away with reusing the existing assembler and that we need a custom one.

3. Also, setting an index for `CuSparseMatrixCSC` is not allowed inside a kernel (ref: https://discourse.julialang.org/t/cuda-jl-atomic-addition-error-to-a-sparse-array-inside-cuda-kernel/78789/2), so I write a very naive `GPUSparseMatrixCSC`.

Indeed, but you should be able to write into nzval directly. Your GPUSparseMatrixCSC struct has very similar structure to the one of CUSPARSE already, so switching should be straight forward.

4. Final observation, in `create_sparsity_pattern` it only create sparse matrix with `nzvals` of type `Float64`  as follows :

    - K = spzeros!!(Float64, I, J, ndofs(dh), ndofs(dh)) #old code
    + K = spzeros!!(T, I, J, ndofs(dh), ndofs(dh)) # my proposal

I don't know whether this was intended or what but I found it worth mentioning.

Indeed. Frekdrik has put something great together to fix this already here #888 and I hope we can merge it in the not so far future to have more direct support of different formats.

Abdelrahman912 · 2024-11-19T18:32:34Z

A GPU benchmark with 1000 X 1000 grid, Biquadratic Lagrange as an approximation function, and 3 x 3 quadrature rule for numerical integration.

Profiler ran for 477.09 ms, capturing 2844 events.
Host-side activity: calling CUDA APIs took 303.82 ms (63.68% of the trace)
┌──────┬───────────┬───────────┬────────┬─────────────────────────┬────────────────────────────┐
│   ID │     Start │      Time │ Thread │ Name                    │                    Details │
├──────┼───────────┼───────────┼────────┼─────────────────────────┼────────────────────────────┤
│    5 │  19.55 µs │   8.82 µs │      1 │ cuMemAllocFromPoolAsync │  15.274 MiB, device memory │
│   19 │  38.39 µs │ 953.67 ns │      1 │ cuStreamSynchronize     │                          - │
│   28 │  44.35 µs │   2.66 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│   33 │   2.71 ms │   7.15 µs │      1 │ cuMemAllocFromPoolAsync │ 244.202 MiB, device memory │
│  170 │   2.84 ms │   1.43 µs │      1 │ cuStreamSynchronize     │                          - │
│  179 │   2.84 ms │  44.51 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│  184 │  47.38 ms │  27.18 µs │      1 │ cuMemAllocFromPoolAsync │ 244.202 MiB, device memory │
│  309 │  47.47 ms │   3.58 µs │      1 │ cuStreamSynchronize     │                          - │
│  318 │  47.49 ms │  41.89 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│  323 │  89.42 ms │  38.86 µs │      1 │ cuMemAllocFromPoolAsync │  15.274 MiB, device memory │
│  335 │  89.48 ms │  64.61 µs │      1 │ cuMemsetD32Async        │                          - │
│  356 │  98.81 ms │  30.76 µs │      1 │ cuMemAllocFromPoolAsync │  34.332 MiB, device memory │
│  370 │  98.86 ms │   4.29 µs │      1 │ cuStreamSynchronize     │                          - │
│  379 │  98.88 ms │    5.4 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│  388 │ 104.32 ms │  21.46 µs │      1 │ cuMemAllocFromPoolAsync │  30.518 MiB, device memory │
│  516 │ 104.41 ms │   2.38 µs │      1 │ cuStreamSynchronize     │                          - │
│  525 │ 104.42 ms │   4.74 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│  534 │ 110.25 ms │  24.56 µs │      1 │ cuMemAllocFromPoolAsync │   3.815 MiB, device memory │
│  548 │ 110.29 ms │   4.53 µs │      1 │ cuStreamSynchronize     │                          - │
│  557 │ 110.31 ms │ 797.99 µs │      1 │ cuMemcpyHtoDAsync       │                          - │
│  566 │ 111.12 ms │  13.35 µs │      1 │ cuMemAllocFromPoolAsync │   7.645 MiB, device memory │
│ 1054 │  111.3 ms │   1.19 µs │      1 │ cuStreamSynchronize     │                          - │
│ 1063 │  111.3 ms │   1.39 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 1072 │ 114.73 ms │  13.59 µs │      1 │ cuMemAllocFromPoolAsync │   3.815 MiB, device memory │
│ 1086 │ 114.75 ms │   1.91 µs │      1 │ cuStreamSynchronize     │                          - │
│ 1095 │ 114.76 ms │ 598.67 µs │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 1112 │ 115.43 ms │   8.11 µs │      1 │ cuMemAllocFromPoolAsync │   3.164 MiB, device memory │
│ 1124 │ 115.44 ms │  50.54 µs │      1 │ cuMemsetD32Async        │                          - │
│ 1129 │ 115.49 ms │   6.91 µs │      1 │ cuMemAllocFromPoolAsync │ 360.000 KiB, device memory │
│ 1141 │  115.5 ms │   6.44 µs │      1 │ cuMemsetD32Async        │                          - │
│ 1162 │ 149.56 ms │   1.22 ms │      1 │ cuMemAllocFromPoolAsync │  34.332 MiB, device memory │
│ 1176 │  150.8 ms │   4.29 µs │      1 │ cuStreamSynchronize     │                          - │
│ 1185 │ 150.82 ms │   7.98 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 1194 │ 158.81 ms │ 722.41 µs │      1 │ cuMemAllocFromPoolAsync │  30.518 MiB, device memory │
│ 1208 │ 159.55 ms │   3.58 µs │      1 │ cuStreamSynchronize     │                          - │
│ 1217 │ 159.56 ms │   6.86 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 1226 │ 167.29 ms │  15.02 µs │      1 │ cuMemAllocFromPoolAsync │   3.815 MiB, device memory │
│ 1240 │ 167.34 ms │   3.58 µs │      1 │ cuStreamSynchronize     │                          - │
│ 1249 │ 167.35 ms │ 777.24 µs │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 1258 │ 168.14 ms │   6.68 µs │      1 │ cuMemAllocFromPoolAsync │   7.645 MiB, device memory │
│ 1965 │ 168.38 ms │   1.19 µs │      1 │ cuStreamSynchronize     │                          - │
│ 1974 │ 168.39 ms │   1.41 ms │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 1983 │ 171.42 ms │  12.87 µs │      1 │ cuMemAllocFromPoolAsync │   3.815 MiB, device memory │
│ 1997 │ 171.46 ms │   2.15 µs │      1 │ cuStreamSynchronize     │                          - │
│ 2006 │ 171.47 ms │ 816.58 µs │      1 │ cuMemcpyHtoDAsync       │                          - │
│ 2023 │ 172.42 ms │   84.4 µs │      1 │ cuLaunchKernel          │                          - │
│ 2801 │ 172.93 ms │  13.11 µs │      2 │ cuMemFreeAsync          │   3.815 MiB, device memory │
│ 2806 │ 172.95 ms │   2.38 µs │      2 │ cuMemFreeAsync          │   7.645 MiB, device memory │
│ 2811 │ 172.96 ms │   2.38 µs │      2 │ cuMemFreeAsync          │   3.815 MiB, device memory │
│ 2816 │ 172.96 ms │   2.62 µs │      2 │ cuMemFreeAsync          │  30.518 MiB, device memory │
│ 2821 │ 172.97 ms │   2.86 µs │      2 │ cuMemFreeAsync          │  34.332 MiB, device memory │
│ 2824 │ 172.97 ms │  303.8 ms │      2 │ cuStreamSynchronize     │                          - │
└──────┴───────────┴───────────┴────────┴─────────────────────────┴────────────────────────────┘

Device-side activity: GPU was busy for 418.87 ms (87.80% of the trace)
┌──────┬───────────┬───────────┬─────────┬────────┬──────┬─────────────┬──────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────
│   ID │     Start │      Time │ Threads │ Blocks │ Regs │        Size │   Throughput │ Name                                                                                            ⋯
├──────┼───────────┼───────────┼─────────┼────────┼──────┼─────────────┼──────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────
│   28 │ 247.24 µs │    2.6 ms │       - │      - │    - │  15.274 MiB │  5.741 GiB/s │ [copy pageable to device memory]                                                                ⋯
│  179 │   3.02 ms │  44.45 ms │       - │      - │    - │ 244.202 MiB │  5.365 GiB/s │ [copy pageable to device memory]                                                                ⋯
│  318 │  47.62 ms │  41.88 ms │       - │      - │    - │ 244.202 MiB │  5.694 GiB/s │ [copy pageable to device memory]                                                                ⋯
│  335 │  89.97 ms │  250.1 µs │       - │      - │    - │  15.274 MiB │ 59.640 GiB/s │ [set device memory]                                                                             ⋯
│  379 │  99.14 ms │   5.27 ms │       - │      - │    - │  34.332 MiB │  6.362 GiB/s │ [copy pageable to device memory]                                                                ⋯
│  525 │ 104.54 ms │   4.76 ms │       - │      - │    - │  30.518 MiB │  6.262 GiB/s │ [copy pageable to device memory]                                                                ⋯
│  557 │ 110.52 ms │ 791.31 µs │       - │      - │    - │   3.815 MiB │  4.708 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 1063 │  111.5 ms │   1.35 ms │       - │      - │    - │   7.645 MiB │  5.519 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 1095 │ 114.97 ms │ 537.16 µs │       - │      - │    - │   3.815 MiB │  6.935 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 1124 │ 115.95 ms │  58.41 µs │       - │      - │    - │   3.164 MiB │ 52.898 GiB/s │ [set device memory]                                                                             ⋯
│ 1141 │ 116.02 ms │  13.11 µs │       - │      - │    - │ 360.000 KiB │ 26.182 GiB/s │ [set device memory]                                                                             ⋯
│ 1185 │ 153.41 ms │   5.53 ms │       - │      - │    - │  34.332 MiB │  6.064 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 1217 │ 161.78 ms │   4.81 ms │       - │      - │    - │  30.518 MiB │  6.200 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 1249 │ 167.66 ms │ 730.51 µs │       - │      - │    - │   3.815 MiB │  5.100 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 1974 │ 168.68 ms │   1.34 ms │       - │      - │    - │   7.645 MiB │  5.582 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 2006 │ 171.79 ms │ 721.93 µs │       - │      - │    - │   3.815 MiB │  5.160 GiB/s │ [copy pageable to device memory]                                                                ⋯
│ 2023 │ 172.92 ms │ 303.78 ms │     256 │     40 │   95 │           - │            - │ assemble_gpu_(CuSparseDeviceMatrixCSC<Float32, Int32, 1l>, CuDeviceArray<Float32, 1l, 1l>, Stat ⋯
└──────┴───────────┴───────────┴─────────┴────────┴──────┴─────────────┴──────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────

KnutAM and others added 10 commits January 11, 2024 10:35

Initial ideas

a979fb2

Working implementation

298158c

Merge branch 'master' into kam/QuadraturePointIterator

1794db3

Add static values version and improve interface

51ab4f2

Add dev example and test

22a7377

Merge branch 'master' into kam/QuadraturePointIterator

18377f3

Add StaticCellValues without stored cell coordinates

27a3a96

initial ideas

95b5729

minor changes

d4e881d

Merge branch 'Ferrite-FEM:master' into cuda-heat-example-w-quaditer

f55b878

koehlerson reviewed May 15, 2024

View reviewed changes

KristofferC reviewed May 15, 2024

View reviewed changes

Abdelrahman912 added 2 commits May 23, 2024 15:54

add some abstractions

c1ef6ad

add minor comment

394ac6a

Abdelrahman912 added 3 commits May 30, 2024 22:57

add z dierction for numerical integration

1f0df67

add Float32

3152042

minor fix

aac5994

Abdelrahman912 commented Jun 4, 2024

View reviewed changes

src/Grid/grid_generators.jl Outdated Show resolved Hide resolved

termi-official mentioned this pull request Jun 11, 2024

CUDA monodomain solver baseline termi-official/Thunderbolt.jl#104

Merged

7 tasks

Abdelrahman912 added 5 commits June 18, 2024 19:52

init coloring implementation

142f89a

init working on the assembler

eaff534

init gpu_assembler

ffdc341

implement naive gpu_assembler

59595e8

minor fix

0e3cb21

Abdelrahman912 added 28 commits October 14, 2024 22:28

try documentation test fix

12f64bb

documentation test fix

361333b

make ci happy

e31c6e3

change kernel launch, init adapt test

626dec2

minor fix

fbc1b4b

add test_adapt, some comments

ea83925

fix precommit

a356d8d

init cpu multi threading

ee1f77c

Merge branch 'master' into cuda-heat-example-w-quaditer

fb7e1fc

hot fix for buggy assembly logic

b38ab72

minor fix

adb166a

test sth

6300a4a

precommit fix

b7301c2

fix explicit imports

18f47b8

add fillzero

f6e9cc6

Merge branch 'master' into cuda-heat-example-w-quaditer

8a796de

minor fix for gpu assembly

75e89ed

minor minor fix

a77c347

make cache mutable

7338788

put the coloring stuff in the init

cbab665

minor fix

1c81281

code for benchmarking (to be removed)

d42bcab

rm cpu multithreading benchmark code

1ab1650

init fix for higher order approximations in gpu

bc8ec95

add working imp for global gpu mem

c7f4b0f

add some comments

d4d5967

trying to make the ci happy

3b2196b

minor fix

825d257

comment gpu related stuff in eg to pass ci

6109bd1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cuda heat example w quaditer #913

Cuda heat example w quaditer #913

Abdelrahman912 commented May 14, 2024

koehlerson May 15, 2024

termi-official May 15, 2024

koehlerson May 15, 2024

termi-official Jun 4, 2024

KristofferC May 15, 2024

Abdelrahman912 commented May 23, 2024

termi-official commented May 23, 2024

Some problems I have encountered that might be so straightforward to tackle:

KnutAM commented May 23, 2024

termi-official commented Jun 4, 2024 •

edited

Loading

Abdelrahman912 commented Jun 20, 2024

termi-official commented Jun 20, 2024

Abdelrahman912 commented Nov 19, 2024

Cuda heat example w quaditer #913

Are you sure you want to change the base?

Cuda heat example w quaditer #913

Conversation

Abdelrahman912 commented May 14, 2024

koehlerson May 15, 2024

Choose a reason for hiding this comment

termi-official May 15, 2024

Choose a reason for hiding this comment

koehlerson May 15, 2024

Choose a reason for hiding this comment

termi-official Jun 4, 2024

Choose a reason for hiding this comment

KristofferC May 15, 2024

Choose a reason for hiding this comment

Abdelrahman912 commented May 23, 2024

What I did for now, and it's still work in progress:

Some problems I have encountered that might be so straightforward to tackle:

termi-official commented May 23, 2024

Some problems I have encountered that might be so straightforward to tackle:

KnutAM commented May 23, 2024

termi-official commented Jun 4, 2024 • edited Loading

Abdelrahman912 commented Jun 20, 2024

So far:

termi-official commented Jun 20, 2024

Abdelrahman912 commented Nov 19, 2024

termi-official commented Jun 4, 2024 •

edited

Loading