Inference: fix batch_size issue.

[xinhaoc]

Description of changes:

Related Issues:

Linked Issues:

• Issue error when I use different batch size for opt inference #846

Issues closed by this PR:

• Closes error when I use different batch size for opt inference #846

Before merging:

• Did you update the flexflow-third-party repo, if modifying any of the Cmake files, the build configs, or the submodules?

[xinhaoc]

@lambda7xx I think our system is correct when using batch_size > 2, something I want to share about that

1. we don't need to change the dimension when create the input tensor, my fault.
2. before running the system with batch_size > 2. we should modify the prompt/test.json, an example is like

["Give three tips for staying healthy.", "Carnegie Mellon Univerity is located in Pittsburgh", "My favorite basketball palyer is Kobe Bryant"]

3. let's fix any following issue you may encounter during the evaluation in this branch.

[jiazhihao]

I am thinking about a more general fix where we make MAX_NUM_REQUESTS, MAX_NUM_TOKENS, and MAX_SEQ_LENGTH input arguments instead of static variables. Following is the Python interface Gabriele and I discussed

from flexflow.serve import LLM, SamplingConfig

llama = LLM.model("decapoda-research/llama-30b-hf", data_type = "half")
ssm1 = LLM.model("Jackframe/llama-160m", data_type = "half")
ssm2 = LLM.model("Jackframe/opt-160m", data_type = "half")

sampling_config = SamplingConfig(temperature = 0.9, topp = 0.8, topk = 1)

LLM.compile(llama, max_parallel_requests = xxx, max_parallel_tokens = yyy, max_seq_length = yyy, tensor_parallel_degree = 4, pipeline_parallel_degree = 2, ssms = {ssm1, ssm2})

result = llama.generate("What's the best xxx in yyy?", sampling = sampling_config)

[xinhaoc]

Yes, that's a good idea.