Discussions

• 3 You must be logged in to vote

  🙏

  How to specify which gpu to use?

  Zhuqln asked Jul 24, 2023 in Q&A · Answered

  

  

  

  

  15
• 1 You must be logged in to vote

  💡

  Can the docker image load a gradio web server?

  mglover-bb started Sep 16, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  💬

  How can I find the specific code implementation involved in KV calculation and writing the calculation results into cache?

  YnRen22852 started Sep 13, 2024 in General

  

  

  0
• 6 You must be logged in to vote

  🙏

  How can I run VLLM serving without an internet connection?

  ZPerling asked Oct 18, 2023 in Q&A · Unanswered

  

  

  

  

  9
• 17 You must be logged in to vote

  💡

  RFC: FP8 in vLLM

  HaiShaw started Jan 17, 2024 in Ideas

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  vLLM failing to recognize GPU from latest official docker image

  DrappierTechnologies asked Jun 19, 2024 in Q&A · Unanswered

  

  

  3
• 2 You must be logged in to vote

  💬

  Structured output with vLLM?

  AkshataDM started Sep 9, 2024 in General

  

  

  

  

  2
• 1 You must be logged in to vote

  🙌

  Distributed inference for re-captioning large datasets with vllm and other things

  sayakpaul started Sep 11, 2024 in Show and tell

  

  

  1
• 18 You must be logged in to vote

  💡

  Support for apple/OpenELM-3B-Instruct

  bks5881 started Apr 25, 2024 in Ideas

  

  

  

  

  3
• 1 You must be logged in to vote

  🙏

  How to get detail metrics when offline llm inference

  tohneecao asked Feb 21, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  LLaVA with GGUF files (and --mmproj argument)?

  benmayersohn asked Sep 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  What do these two functions do

  daliwang777 started Sep 9, 2024 in General

  

  

  0
• 2 You must be logged in to vote

  💡

  The Phenomenon of Latency Jumps in Inference

  daliwang777 started Sep 5, 2024 in Ideas

  

  

  2
• 1 You must be logged in to vote

  🙏

  Why do vllm set default keep-alive timeout to 5s?

  SeungminHeo asked Sep 9, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💡

  Can Paged Attention be used in training to reduce memory usage?

  CACACAZ started Sep 9, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  💬

  how to fix "Engine iteration timed out. This should never happen"

  liutao053877 started Sep 7, 2024 in General

  

  

  0
• 3 You must be logged in to vote

  💬

  Distributed configuration for LLama 3.1 70b FP8 and 4 h100 GPUs server

  wakusoftware started Aug 29, 2024 in General

  

  

  1
• 2 You must be logged in to vote

  🙏

  How to enable embeddings mode

  mina58 asked Sep 3, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  In vLLM v0.5.5, does --num-scheduler-steps not work with --enable-lora?

  ashgold asked Sep 2, 2024 in Q&A · Closed · Answered

  

  

  2
• 3 You must be logged in to vote

  🙏

  pip install the CPU version?

  johann-petrak asked Sep 2, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Estimate GPU Type or Total VRAM Required using HF Repo ID

  stikkireddy asked Sep 2, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Question on ThreadPoolExecutor and GIL Impact

  satshiv7983 asked Sep 2, 2024 in Q&A · Unanswered

  

  

  0
• 0 You must be logged in to vote

  🙏

  Dynamic Model Loading with Docker Deployment

  ArijitSinghEDA asked Sep 2, 2024 in Q&A · Unanswered

  

  

  0
• 20 You must be logged in to vote

  💡

  /v1/embeddings please

  feature request

  yuhai-china started Jun 21, 2023 in Ideas

  

  

  

  

  27
• 2 You must be logged in to vote

  🙏

  Do you support streaming generating outputs?

  question Further information is requested

  ltz0120 asked Jun 24, 2023 in Q&A · Answered

  

  

  

  

  6