NWC good first issues (Cuda)

External fp8 baseline: vLLM (fp8 weight-only) tokens/s on the same GPUs about 1 month ago

help wanted roadmap benchmark

2

Neural Weight Compression: lossless BF16 weights (-31 %) with a fused CUDA matvec that decodes in registers, faster than cuBLAS on RTX 4070 and NVIDIA A16, bit-exact

Cuda

#bf16#compression#cuda#gpu#inference#llm#llm-inference#lossless#lossless-compression#nvidia#pytorch#transformers#weight-compression

Ada (RTX 4070) needs ~17 SM clocks per warp step where Ampere (A16) needs 11, same SASS about 1 month ago

help wanted roadmap

parda21/NWC

2

Neural Weight Compression: lossless BF16 weights (-31 %) with a fused CUDA matvec that decodes in registers, faster than cuBLAS on RTX 4070 and NVIDIA A16, bit-exact

Cuda

#bf16#compression#cuda#gpu#inference#llm#llm-inference#lossless#lossless-compression#nvidia#pytorch#transformers#weight-compression

Benchmark results wanted: post your GPU here about 1 month ago

help wanted benchmark

parda21/NWC

2

Neural Weight Compression: lossless BF16 weights (-31 %) with a fused CUDA matvec that decodes in registers, faster than cuBLAS on RTX 4070 and NVIDIA A16, bit-exact

Cuda

#bf16#compression#cuda#gpu#inference#llm#llm-inference#lossless#lossless-compression#nvidia#pytorch#transformers#weight-compression

llama.cpp port: GGML tensor type, CUDA mmv kernel, converter about 1 month ago

help wanted roadmap

parda21/NWC

2

Neural Weight Compression: lossless BF16 weights (-31 %) with a fused CUDA matvec that decodes in registers, faster than cuBLAS on RTX 4070 and NVIDIA A16, bit-exact

Cuda

#bf16#compression#cuda#gpu#inference#llm#llm-inference#lossless#lossless-compression#nvidia#pytorch#transformers#weight-compression

Colab notebook on a free T4 (sm_75) about 1 month ago

help wanted roadmap

parda21/NWC

2

Neural Weight Compression: lossless BF16 weights (-31 %) with a fused CUDA matvec that decodes in registers, faster than cuBLAS on RTX 4070 and NVIDIA A16, bit-exact

Cuda

#bf16#compression#cuda#gpu#inference#llm#llm-inference#lossless#lossless-compression#nvidia#pytorch#transformers#weight-compression

Measure H100 PCIe and H100 SXM about 1 month ago

help wanted roadmap benchmark

parda21/NWC

2

Neural Weight Compression: lossless BF16 weights (-31 %) with a fused CUDA matvec that decodes in registers, faster than cuBLAS on RTX 4070 and NVIDIA A16, bit-exact

Cuda

#bf16#compression#cuda#gpu#inference#llm#llm-inference#lossless#lossless-compression#nvidia#pytorch#transformers#weight-compression

parda21/NWC

Open Issues Need Help