Skip to content

Commit

Permalink
Update README.md to be more detailed
Browse files Browse the repository at this point in the history
  • Loading branch information
jshuadvd committed Jul 16, 2024
1 parent dfba729 commit 7583b17
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ The **LongRoPE** model architecture is designed to extend the context window of

The LongRoPE model extends the context window of large language models beyond 2 million tokens. Key components include:

"1." Rotary Position Encoding (RoPE):
1. Rotary Position Encoding (RoPE):

```python
class RoPEPositionalEncoding(nn.Module):
Expand All @@ -86,7 +86,7 @@ The LongRoPE model extends the context window of large language models beyond 2
return sin_cos.view(*sin_cos.shape[:-2], -1)
```

"2." Non-uniform Interpolation:
2. Non-uniform Interpolation:

```python
def non_uniform_interpolation(pos_embed, extension_ratio, lambda_factors, n_hat):
Expand All @@ -101,7 +101,7 @@ The LongRoPE model extends the context window of large language models beyond 2
return interpolated_pos
```

"3." Progressive Extension Strategy:
3. Progressive Extension Strategy:

```python

Expand Down

0 comments on commit 7583b17

Please sign in to comment.