Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom AllReduce #1467

Merged
merged 54 commits into from
Aug 15, 2024
Merged

Custom AllReduce #1467

merged 54 commits into from
Aug 15, 2024

Conversation

chenzhuofu
Copy link
Collaborator

@chenzhuofu chenzhuofu commented Aug 14, 2024 •
edited
Loading

Description of changes:

  • Support custom AllReduce in the tensor parallelism.
  • Use bit-wise mask in flasinfer.
  • Format the code.

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

This change is Reviewable

chenzhuofu and others added 30 commits June 29, 2024 02:06
@chenzhuofu
feat: update deps/flashinfer
7e02ef0
@chenzhuofu
feat: update flashinfer
be64b3b
@chenzhuofu
fix: now can get correct result, but has performance problem
7a4543b
@chenzhuofu
fix: update_custom_mask performance
e1c8145
@chenzhuofu
chore: minor
483b71b
@chenzhuofu
chore: add perf code
0a117fb
@zikun-li
Merge branch 'specscheduler' into specscheduler-new-attention
ac343ee
@chenzhuofu
feat: add attention metadata
3e9dc00
@chenzhuofu
feat: add AttentionMetaData
a498e6f
@chenzhuofu
feat: tree_verify_attn use global attentionmetadata
2903996
@chenzhuofu
feat: move attentionmetasize to global computing
fed9e5c
@chenzhuofu
chore: minor
7c64f36
@chenzhuofu
chore: remove unused
37d3e3d
@chenzhuofu
feat: add spec_inc_attn backup
9940ff5
@chenzhuofu
feat: SSM use flashinfer kernel
5262af7
@chenzhuofu
fix: SSM don't use cudaGraph
dfe4bec
@chenzhuofu
chore: remove redundant code
e08f06e
@chenzhuofu
chore: comment out minor
ed544e9
@chenzhuofu
feat: attention adapt to cudaGraph
214aed6
@chenzhuofu
fix: split handler_collections for prompt/decode phases
020129b
@chenzhuofu
chore: tree verify cannot use cudaGraph
0150501
@chenzhuofu
feat: move all flashinfer-related states to global (tree search atten…
d4d66af 
…tion)
@chenzhuofu
fix: use identical attention_meta instance across all FFHandlers
b4cc53b
@chenzhuofu
feat: enable cudaGraph in tree search mode
79ea1d9
@chenzhuofu
chore: minor
8d52e4f
@chenzhuofu
feat: tree search & verify use separate attention_meta
2afb66c
@chenzhuofu
fix: attention_metadata should be distinct for each worker
74e166f
@chenzhuofu
feat: tree verify attention use metadata
252a5c4
@chenzhuofu
feat: support llm cudaGraph
9fda3b6
@chenzhuofu
chore: minor
6f88134
@chenzhuofu
chore: minor
0e5ec41
@chenzhuofu
fix: llm cudaGraph, should ensure the kernel parameter be consistent
dd4d6d0
@chenzhuofu
feat: reduce cudaGraph number
18fee1c
@chenzhuofu
feat: reduce cudaGraph instances number
b17c5cb
@chenzhuofu
feat: add tensorRT-LLM custom_allreduce
428875c
@chenzhuofu
feat: add tensorrt_llm custom_allreduce kernel into exeutable
840da50
@chenzhuofu
doc: add a README for acknowledgement
5f16bd4
@chenzhuofu
feat: add device info in FFHandle
3d50053
@chenzhuofu
feat: enable both cudaGraph
273dfc7
@chenzhuofu
feat: temporally add the ipc mem
d201b2e
@chenzhuofu
feat: enable only ssm cudaGraph
1ba38a8
@chenzhuofu
feat: minor reconstruct
7b235ef
@chenzhuofu
feat: implementation of CommunicationBuffer
f3c9629
@chenzhuofu
feat: implement custom_allreduce
462e0b7
@chenzhuofu
feat: allocate memory from legion, not cudaMalloc
567e165
@chenzhuofu
chore: some debug output
a2fb367
@chenzhuofu
feat: switch to use peer memory, rather than IPC memory
4134013
@chenzhuofu
chore: remove debug output
c094981
@chenzhuofu
fix: minor concurrent bug
95ca71c
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
55a7942 
…cscheduler-new-attention
@chenzhuofu
style: format code
b9760c0
@jiazhihao jiazhihao merged commit 2b02648 into specscheduler Aug 15, 2024
36 of 39 checks passed
@chenzhuofu chenzhuofu deleted the specscheduler-new-attention branch August 15, 2024 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiazhihao jiazhihao jiazhihao approved these changes

@zikun-li zikun-li zikun-li approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chenzhuofu @jiazhihao @zikun-li