poojithdevan4D

poojithdevan4D/pooji-vllm

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

0 stars 0 forks 0 watchers Python Apache License 2.0
cuda inference kv-cache llm pytorch vllm
View on GitHub
6 Open Issues Need Help Last updated: Aug 23, 2026

Open Issues Need Help

View All on GitHub
Add an OpenAI-compatible /v1/chat/completions endpoint about 7 hours ago
good first issue
poojithdevan4D/pooji-vllm
0

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

Python
#cuda#inference#kv-cache#llm#pytorch#vllm
Working INT8/INT4 weight-only quantization about 9 hours ago
help wanted performance
poojithdevan4D/pooji-vllm
0

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

Python
#cuda#inference#kv-cache#llm#pytorch#vllm
Chunked prefill (stop TTFT growing with queue position) about 9 hours ago
help wanted performance
poojithdevan4D/pooji-vllm
0

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

Python
#cuda#inference#kv-cache#llm#pytorch#vllm
Fused paged attention kernel (remove the per-layer K/V gather) about 9 hours ago
help wanted performance
poojithdevan4D/pooji-vllm
0

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

Python
#cuda#inference#kv-cache#llm#pytorch#vllm
Swap-to-CPU preemption as an alternative to recompute about 9 hours ago
help wanted
poojithdevan4D/pooji-vllm
0

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

Python
#cuda#inference#kv-cache#llm#pytorch#vllm
Support more model architectures about 9 hours ago
good first issue
poojithdevan4D/pooji-vllm
0

A vLLM-style LLM inference engine written from scratch on a 4 GB laptop GPU: paged KV cache, continuous batching, prefix caching, preemption, CUDA graphs, and a streaming server. 3.65x HF generate() on realistic workloads.

Python
#cuda#inference#kv-cache#llm#pytorch#vllm