llm-d-router good first issues (Go)

[CI] e2e image builds use the live PR merge ref while tests deploy the pinned checkout 42 minutes ago

help wanted needs-triage kind/bug

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Flow Control] Make fairness and ordering agentic-aware about 12 hours ago

help wanted area/flowcontrol

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Enable forced endoint selection by EPP 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Enable external control over EPP selection 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Dynamic LoRA placement/balancing in InferencePool (beyond “route to where it’s loaded”) 1 day ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Feature Request] A Prefetch Plugin for Reducing latency when reading KV cache from disk in Tiered Storage 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Refactor the Pod watching to fit the Discovery plugin 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Make it easier to add new features to the latency predictor 1 day ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Review sync.Map usage across codebase 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

token-producer: explicit backend selector field 1 day ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Performance optimizations for vllm render api in precise prefix-cache aware routing for multimidal workloads 1 day ago

help wanted needs-triage area/mm

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Make speculative prefix-cache indexing group-aware for HMA models 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

feat(datalayer): extend `file-discovery` to support cluster endpoints 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

feat(epp): hub mode for multi-cluster routing 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

approx/estimate prefix producer never indexes generated (decode) tokens 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

expose KV cache location (device tier) in prefix-cache match data 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Wire P2P prefix pull (x-kv-cache-source-host-port) through the encode->PD path 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Coordinator] Check reuse sidecar KV connectors 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Feature] Hot-reload the metrics/models scrape client cert on rotation 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[SGLang] Forward selected DP ranks in P/D requests 1 day ago

help wanted needs-triage sglang

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[RFC] llm-d-router Profiler 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Flow Control] Avoid Head of Line Blocking from new Sessions in Agentic Scenarios 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

EPP precise prefix cache: request-path CPU and allocations at fleet scale 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Flow Control] Enforce queue-wait budgets from the expiry sweep only 1 day ago

help wanted area/flowcontrol

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Accurate Cached Token Usage Metrics in P/D Setup 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Flow Control] Response-feedback contract for external dispatchers (tracking) 1 day ago

help wanted area/flowcontrol

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Coordinator] Support Responses API `previous_response_id` via agentic-api 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Coordinator] Serve Responses turns that use server-side tool calling 1 day ago

help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Precise prefix matching: separation of concerns and refactor plan 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

kvblock: InMemoryIndex.Lookup tests emptiness and reads entries without the pod-cache lock 1 day ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Improve datalayer observability 8 days ago

help wanted area/telemetry

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

coordinator: help wanted - run the coordinator on non-NVIDIA accelerators 10 days ago

help wanted needs-triage kind/feature area/coordinator

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[p2p-source-producer] Enable cost-aware pull decision (Child issue #1 of Issue #2273) 25 days ago

good first issue area/scheduling area/kvcache

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

prefix-cache-affinity-filter: TTFT load gate compares observed values against substituted defaults when an endpoint lacks its load attribute about 1 month ago

good first issue kind/bug area/scheduling

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Compatibility] Validate and document TurboQuant (KV cache quantization) serving support about 1 month ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[E2E Tests] Use standalone mode for e2e tests about 1 month ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Support enabling EPP feature gates without duplicating the full plugin config about 2 months ago

good first issue area/dev

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Remove deprecated dp_profile_handler 3 months ago

help wanted lifecycle/rotten

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Logs are not correlated with traces: trace_id/span_id never injected into the logger 3 months ago

help wanted needs-triage kind/bug observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Refactor label-based filters 3 months ago

help wanted lifecycle/rotten

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Add span for filter 3 months ago

help wanted observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Deprecate load-aware scorer 3 months ago

help wanted lifecycle/stale

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[E2E Tests] Failing on a few PRs 3 months ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for mm-embeddings-cache-producer plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for program-aware-fairness plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for precise-prefix-cache-producer plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Add span for picker 4 months ago

help wanted observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Migrate e2e tests from GAIE 4 months ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for approx-prefix-cache-producer plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Add pluginName to the default DumpState 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for predicted-latency-producer plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for file-discovery plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Implement DumpState for no-hit-lru-scorer plugin 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Operations] Tracking: implement DumpState for all in-tree plugins 4 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Observability] An EPP metric that reports inflight tokens and requests 4 months ago

help wanted needs-triage observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Observaibility] epp/metrics: unbounded label cardinality from user-controlled inputs (metrics cardinality DoS) 4 months ago

help wanted needs-triage kind/bug observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Observability] P/D sidecar observability: add Prometheus /metrics, wire trace propagation, document enablement 4 months ago

help wanted needs-triage observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

A session-affinity-filter plugin 4 months ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Observability] Add fairness_id and objective labels to EPP TTFT/TPOT metrics for per-tenant SLO observability 5 months ago

help wanted observability

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

more granular request-based load-aware scorers 5 months ago

good first issue help wanted needs-triage

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Review and types of fix CRD fields 5 months ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[migration] GAIE changes and migration 5 months ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

EPP should signal invalid plugin config fields via error/warning 5 months ago

good first issue help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

Add helm charts builds to ci 5 months ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Migration] rename all headers managed by epp to have the prefix x-llm-d 5 months ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Migration] Remove the InferencePool CRD definitions. 5 months ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Migration]: Migrate the InferenceObjective and InferenceModelRewrite APIs to llm-d.ai group name 5 months ago

help wanted triage/accepted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

[Migration]: rename the docker images 5 months ago

help wanted

llm-d/llm-d-router

236

llm-d Router: The intelligent entry point for inference requests

Go

#ai#gateway-api#inference#kubernetes#networking

llm-d/llm-d-router

Open Issues Need Help