vllm-project

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

85.0K stars 18.8K forks 85.0K watchers Python Apache License 2.0
amd blackwell cuda deepseek deepseek-v3 gpt gpt-oss inference kimi llama llm llm-serving model-serving moe openai pytorch qwen qwen3 tpu transformer
View on GitHub Website
100 Open Issues Need Help Last updated: Jul 1, 2026

Open Issues Need Help

View All on GitHub
[Bug]: ModelOpt Llama-4 Checkpoints Take 5+ minutes to load about 12 hours ago
bug help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
Upgrade to Transformers v5 1 day ago
help wanted
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Enable LoRA support for tower and connector in more MM models 2 days ago
help wanted feature request multi-modality
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Performance]: qknorm+rope fusion slower than unfused on H100 2 days ago
help wanted performance torch.compile stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Priority scheduling supports preemption of requests in the running queue by requests in the waiting queue 4 days ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug][RL]: Port Conflict 5 days ago
bug help wanted good first issue unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Batch Invariant Feature and Performance Optimization 7 days ago
good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Improve `--kv-cache-dtype` behavior when checkpoint specifies `kv_cache_quant_algo` 8 days ago
bug good first issue unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] InternVL2 9 days ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Roadmap]: PD Disaggregation with `NixlConnector` Roadmap 9 days ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] Tarsier2ForConditionalGeneration 10 days ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[RFC]: Support ViT Full CUDA Graph (Tracker) 10 days ago
help wanted RFC multi-modality
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Integrate fused `kMoEFinalizeARResidualRMSNorm` from FlashInfer 13 days ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[torch.compile] config hashing refactor follow-ups 14 days ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Improve DCP error messages 14 days ago
good first issue feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Tracking Whisper feature requests 14 days ago
help wanted good first issue feature request keep-open multi-modality
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Extract KV-Cache update from all attention backends 14 days ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Error Logging Redesign 16 days ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[RFC]: Improve environment variable declaration and handling 18 days ago
help wanted good first issue RFC keep-open
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Composite model loading using `AutoWeightsLoader` for all models 19 days ago
good first issue feature request keep-open
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Refactor Int8ScaledMMLinearLayerConfig to use QuantKey 21 days ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Docs] Document NIXL KV connector metrics aggregation semantics 27 days ago
good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[torch.compile][Performance]: Unwrap custom ops and improve fusion (Inductor and custom) 27 days ago
help wanted performance torch.compile stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Mistral Small 4 EAGLE drafter crashes at init — EagleMistralLarge3Model missing use_mha (skips DeepseekV2Model.__init__) 28 days ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Remove attention layer name from `unified_kv_cache_update` 28 days ago
help wanted good first issue feature request torch.compile unstale startup-ux
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Certain Ranks Take a Look Time to Load Weights 29 days ago
bug help wanted
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Optimize collectives in TP MoE case using torch.compile pass 29 days ago
help wanted good first issue performance feature request torch.compile
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: torch.compile padding causes IMA on Hopper + DBO about 1 month ago
bug help wanted torch.compile stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[torch.compile] E2E correctness testing for fusions about 1 month ago
help wanted torch.compile
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Better Flashinfer compilation logging about 1 month ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] IsaacForConditionalGeneration about 1 month ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: `CustomOp` cleanup about 1 month ago
help wanted good first issue feature request keep-open startup-ux
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] Base model and LoRA used in test has incorrect `tokenizer_config.json` about 1 month ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: FlashInfer Incompatible with Sleep Mode about 1 month ago
bug help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Unify MoE "Oracles" with Class Structure about 1 month ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Performance]: Update Cascade Attention Heuristics for FA3 about 1 month ago
help wanted good first issue feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Support `Phi4Flash` model in V1 about 1 month ago
help wanted good first issue feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[RFC]: Limit the use of envvars in vLLM about 1 month ago
good first issue RFC
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Support DCP with FP8 KV Cache about 1 month ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Implement Concurrent Partial Prefills In V1 Engine about 1 month ago
help wanted feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Fused Kernel for GPT-OSS Router about 2 months ago
help wanted good first issue feature request stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Default eplb num_redundant_experts to the lowest valid value if unspecified about 2 months ago
help wanted good first issue feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature][Cleanup]: Unify `vllm.utils.flashinfer` and `vllm.model_executor.layers.quantization.utils.flashinfer_utils` about 2 months ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug][Help Wanted]: PPLX + vLLM CUTLASS FP8 Gives Incorrect Responses about 2 months ago
bug help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Usage]: How to get query embeddings from ColBERT? about 2 months ago
good first issue usage
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Integrate RMS+fp4 fused kernel from FlashInfer about 2 months ago
help wanted good first issue feature request torch.compile stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Use kernel abstraction for fp4 scaled mm 2 months ago
help wanted feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Add tests for all parallel sampling parameter combinations 2 months ago
bug good first issue keep-open stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Add Triton implementation of NVFP4 GEMM 2 months ago
good first issue performance feature request unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[bug] `spawn_new_process_for_each_test` decorator broken 2 months ago
bug help wanted ci/build
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[CI Failure]: Fusion E2E TP2 Quick (H100) 2 months ago
help wanted torch.compile ci-failure needs reproduction
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Wrong timestamps if audio > 30s 2 months ago
bug help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Performance]: Compiled `QuantFP8.forward_native` group quantization (1, 128) slower than CUDA on H100/RTX5090 2 months ago
help wanted performance torch.compile keep-open needs reproduction
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Hangs on mulit-node setup when len(prompt) > max_model_len 2 months ago
bug good first issue stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: KV Cache grows continuously with just one chat completion request using meta-llama/Llama-3.2-1B on L40 GPU with Flash Attention and finally completed after 10 minutes 3 months ago
bug help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: 0% Acceptance rate with FI Cutlass DeepSeekR1 NVFP4 with mtp ep 3 months ago
bug help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Upstream DGX spark improvements from Avarok-Cybersecurity/dgx-vllm 3 months ago
help wanted feature request nvidia quantization
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Worse EAGLE3 acceptance rates on MRV2 3 months ago
bug help wanted
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[CI Failure]: PyTorch Compilation Passes Unit Tests 3 months ago
help wanted torch.compile ci-failure
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] NemotronParseForConditionalGeneration 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
Automatically select highest priority batch-invariant attention backend 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Add LoRA support for Gemma4ForConditionalGeneration / Gemma 4 models 3 months ago
good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Add support for fused fp8 output to FlashAttention 3 3 months ago
help wanted performance feature request torch.compile keep-open
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Tracker]: Use `mm_features` for M-RoPE calculation for all models 3 months ago
help wanted multi-modality
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] Distributed shutdown test timetout 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Remove MXFP4 Logic From `fused_experts` 3 months ago
help wanted good first issue feature request stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Qwen3 Omni Transcriptions 3 months ago
good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Run performance benchmarks for multi-modal models in CI 3 months ago
help wanted feature request keep-open multi-modality
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] SarvamMLAForCausalLM 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] Ernie4_5_VLMoeForConditionalGenerati 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] MiniCPMV cannot apply processor 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] ColBERTJinaRobertaModel 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] HCXVisionForCausalLM 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Transformers v5] Mistral multimodal models 3 months ago
help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Make Whisper model compatible with cuda graphs 3 months ago
bug help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Consolidate GPTQ Quantization 4 months ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: Qwen3-VL 2:4 sparsity llm-compressor RuntimeError: shape mismatch (0.12, 0.13rc2) 4 months ago
bug help wanted good first issue stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug] DCP/DBO: 'NoneType' error building attention_metadata during DeepSeek-V3.1 deployment dummy run 4 months ago
bug help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Add LoRA support for Qwen3ASRForConditionalGeneration 4 months ago
good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Refactor `aiter_shared_expert_fusion` 4 months ago
help wanted stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: JAIS: ALiBi is applied even when position_embedding_type="learned" 4 months ago
bug good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: `vllm bench serve` isn't an exact replacement of benchmark_serving.py 4 months ago
bug good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature][Help Wanted]: Add tuning script and config files for Mamba selective_state_update kernel 4 months ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: torch.opcheck fails for `_C.rms_norm_per_block_quant` 4 months ago
bug help wanted
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug][UX]: Unclean shutdown from ctrl-c with AR Fusion 4 months ago
bug help wanted good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: matryoshka need gpu-memory??? 4 months ago
bug good first issue
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Remove Chunking From FusedMoE 4 months ago
help wanted good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: `swap_space` parameter is unused and useless 4 months ago
bug good first issue keep-open
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: VLLM_DISABLE_COMPILE_CACHE should be a config flag 4 months ago
help wanted feature request torch.compile stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: AsyncTP pass has poor perf on B200 4 months ago
bug help wanted torch.compile keep-open
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Qwen-ASR Forced Aligner 4 months ago
good first issue feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[BugFix]: move torch.Size across graphs in split_graph 4 months ago
help wanted feature request torch.compile
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Better base64 to torch tenser 4 months ago
good first issue feature request stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Implement naive prepare/finalize class to replace naive dispatching in fused_moe/layer.py 4 months ago
help wanted good first issue feature request stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Disable logging `/metrics` 5 months ago
help wanted good first issue feature request stale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Support norm+quant & silu+quant fusion for block (group) quantization 5 months ago
help wanted good first issue feature request torch.compile
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Improve config validation using Pydantic 5 months ago
good first issue keep-open
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Misc]: CMake Clean-up / Refactor Tasks 5 months ago
help wanted good first issue keep-open unstale
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Bug]: QKNorm+RoPE fusion broken for qwen3-fp8 on B200 5 months ago
bug help wanted torch.compile
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer
[Feature]: Benchmark torch._scaled_mm performance with and without padding 5 months ago
help wanted feature request
vllm-project/vllm
85.0K

A high-throughput and memory-efficient inference and serving engine for LLMs

Python
#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference#kimi#llama#llm#llm-serving#model-serving#moe#openai#pytorch#qwen#qwen3#tpu#transformer