llm-d/llm-d

[CI/CD] Add Nightly P/D test for OCP 2 days ago

help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Doc]: Better Document Operational Flows 3 days ago

documentation good first issue help wanted release/v0.9

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[CI/CD] Nightly matrix health tracker — May 2026 3 days ago

help wanted CI/CD release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[CI/CD] nightly-e2e-predicted-latency-ocp references non-existent reusable workflow 3 days ago

help wanted CI/CD release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Align repository with llm-d repo template (7 items) 14 days ago

help wanted lifecycle/stale

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[P/D] Support Mooncake Connector 15 days ago

enhancement help wanted release/v0.9

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Batch] Graduate Async Processor to Production 20 days ago

help wanted release/v0.9

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Guides] update all guides to properly configure request draining 20 days ago

good first issue help wanted release/v0.9

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Docs]: A guide for deploying multiple models 20 days ago

good first issue help wanted release/v0.9

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Document precise prefix-cache routing limitations 22 days ago

help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Autoscaling] Add a guide to show WVA ability to balance between two different InferencePools about 1 month ago

help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Batch] Graduate Batch Gateway to Production about 1 month ago

enhancement help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Router] Create Multi-Modal Guide about 1 month ago

help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Install] Migrate llm-d-router Helm Charts to llm-d from GAIE about 2 months ago

help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[CI/CD] OCP nightlies fail with "Pods did not become ready within 30m" about 2 months ago

help wanted CI/CD release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Router] Responses and Messages API about 2 months ago

enhancement help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Docs]: Document APIs supported by llm-d about 2 months ago

documentation help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[refactor] refactor E2E approach about 2 months ago

help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Graduation]: `llm-d-infra` should be graduated from the `llm-d-incubation` to `llm-d` org about 2 months ago

documentation enhancement good first issue help wanted hacktoberfest-accepted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Docs] Consolidate Gateway Guides/Prereqs about 2 months ago

good first issue help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Docs]: Guides to demonstrate various rollouts scenarios about 2 months ago

documentation good first issue help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Docs] Streamline Monitoring Setup about 2 months ago

documentation help wanted release/v0.8

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Align guides with llm-d Router terminology about 2 months ago

help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Create an issue template for new hardware and platform support 3 months ago

documentation good first issue CI/CD lifecycle/rotten

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Feat] Buildup regression testing infrastructure 3 months ago

help wanted CI/CD lifecycle/rotten

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Feature]: Helpers for Deploying Profiling Tools 3 months ago

enhancement help wanted lifecycle/rotten

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Configure best practice lifecyle hooks and probes in vllm deployments 3 months ago

enhancement good first issue official-guides lifecycle/rotten

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[enhance] Optimizing Inference Image Size 3 months ago

good first issue CI/CD Image builds hacktoberfest-accepted lifecycle/rotten

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Demo] Multiple model deployment 3 months ago

enhancement good first issue help wanted hacktoberfest-accepted lifecycle/rotten

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Feat] Upgrade NIXL to v0.6.1 3 months ago

good first issue Image builds hacktoberfest-accepted lifecycle/stale

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Update LMCache version in llm-d image 8 months ago

good first issue help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Inference image Readiness Probe 9 months ago

good first issue fixed - pending user validation Image builds

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

[Bug]: LMCache fails when using LMCacheConnectorV1 with ImportError 9 months ago

bug help wanted

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Add GKE internal LB gateway to user guides 10 months ago

good first issue guides

llm-d/llm-d

3.6K

Achieve state of the art inference performance with modern accelerators on Kubernetes

Shell

#ai#cncf#distributed-inference#gpu#inference#intelligent-routing#kubernetes#llm#model-server

Open Issues Need Help