[SpecInfer] Update RequestManager

[zwang86]

Description of changes:

Related Issues:

Linked Issues:

• Issue #

Issues closed by this PR:

• Closes #

[zwang86]

It seems we still have some issues with longer prompt sets in the spec_beam_attention kernel. I am getting an CUDA error with these prompt set:

["Write a detailed product description for a food chopper tool that lets you chop fruits and vegetables.",
    "Write a short blog post (500 words) about the best dog toys for new dog owners.",
    "ChatGPT is rewriting Genesis.",
    "Please write the evolution of humans by natural selection in the form of a recipe."]

 ** On entry to GEMM_EX  parameter number 18 had an illegal value
 ** On entry to GEMM_EX  parameter number 18 had an illegal value
Cuda failure: 7
/home/zeyuwang/FlexFlow/src/ops/spec_inc_multihead_self_attention.cu:316
Aborting...
spec_infer: /home/zeyuwang/FlexFlow/src/ops/spec_inc_multihead_self_attention.cu:316: void FlexFlow::Kernels::SpecIncMultiHeadAttention::compute_attention_kernel(const FlexFlow::SpecIncMultiHeadSelfAttentionMeta*, const FlexFlow::BeamSearchBatchConfig*, int, DT*, const DT*, const DT*, cudaStream_t) [with DT = __half; cudaStream_t = CUstream_st*]: Assertion `false' failed.
Aborted (core dumped)

[zwang86]

The issue

It seems we still have some issues with longer prompt sets in the spec_beam_attention kernel. I am getting an CUDA error with these prompt set:

["Write a detailed product description for a food chopper tool that lets you chop fruits and vegetables.",
    "Write a short blog post (500 words) about the best dog toys for new dog owners.",
    "ChatGPT is rewriting Genesis.",
    "Please write the evolution of humans by natural selection in the form of a recipe."]

 ** On entry to GEMM_EX  parameter number 18 had an illegal value
 ** On entry to GEMM_EX  parameter number 18 had an illegal value
Cuda failure: 7
/home/zeyuwang/FlexFlow/src/ops/spec_inc_multihead_self_attention.cu:316
Aborting...
spec_infer: /home/zeyuwang/FlexFlow/src/ops/spec_inc_multihead_self_attention.cu:316: void FlexFlow::Kernels::SpecIncMultiHeadAttention::compute_attention_kernel(const FlexFlow::SpecIncMultiHeadSelfAttentionMeta*, const FlexFlow::BeamSearchBatchConfig*, int, DT*, const DT*, const DT*, cudaStream_t) [with DT = __half; cudaStream_t = CUstream_st*]: Assertion `false' failed.
Aborted (core dumped)

The issue can be solved by applying a similar change as we discussed above.
Due to skipping attention computation for the pending request, this assertion no longer holds, but the out puts seems reasonable while commenting out the assertion. @xinhaoc Do you think we need to make any modifications in the attention kernel?
Note: although the code is located in spec_inc_multihead_self_attention, the func is actually called in verify_inc_multihead_self_attention.