Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
Signed-off-by: Rui Campos <mail@ruicampos.org>
  • Loading branch information
RuiFilipeCampos authored Mar 21, 2024
1 parent 7ebe808 commit 2695c26
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ $$q^{nul} _ {l} = \delta_{f(l)g(l)} \bar M^n_{l} \left [ \delta^{f(u)g(u)} p^{nf

which we'll now group according to the $\delta$'s

$$q^{nul} _ {l} = \bar M^n _ {l} p^{nf(u)f(l)} p^{nf(u)f(l)} \delta_{f(l)g(l)} \delta^{f(u)g(u)} + 2 \bar M^n_{l} p^{nf(u)f(l)} p^{ng(u)f(l)} \delta_{f(l)g(l)} \tilde \delta^{f(u)g(u)} + 2 \bar M^n_l p^{nf(u)f(l)} p^{nf(u)g(l)} \delta^{f(u)g(u)} \tilde \delta_{f(l)g(l)} + 4 \bar M^n_l p^{nf(u)f(l)} p^{ng(u)g(l)} \tilde \delta^{f(u)g(u)} \tilde \delta_{f(l)g(l)}$$
$$q^{nul} _ {l} = \bar M^n _ {l} p^{nf(u)f(l)} p^{nf(u)f(l)} \delta^{f(l)g(l)} \delta^{f(u)g(u)} + 2 \bar M^n_{l} p^{nf(u)f(l)} p^{ng(u)f(l)} \delta^{f(l)g(l)} \tilde \delta^{f(u)g(u)} + 2 \bar M^n_l p^{nf(u)f(l)} p^{nf(u)g(l)} \delta^{f(u)g(u)} \tilde \delta^{f(l)g(l)} + 4 \bar M^n_l p^{nf(u)f(l)} p^{ng(u)g(l)} \tilde \delta^{f(u)g(u)} \tilde \delta^{f(l)g(l)}$$

For a given combination of $u$ and $l$, there's only one term to be calculated. All terms will be computed in paralel on the gpu and collected onto a tensor that represents $q^{nul} _ {l}$, as demanded by the tensor notation, a sum is then performed over $l$ to obtain $q^{nu}$. The lookup tables for $f$ and $g$ are then used to recover $q^{ncc'}$ which is then comunicated back to torch for the rest of the attention mechanism.

Expand Down

0 comments on commit 2695c26

Please sign in to comment.