withcatai/node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level

1.6K stars 0 forks 0 watchers TypeScript

ai bindings catai cmake cmake-js cuda embedding function-calling gguf gpu grammar json-schema llama llama-cpp llm metal nodejs prebuilt-binaries self-hosted vulkan

View on GitHub

1 Open Issue Need Help Last updated: Jul 4, 2025

Open Issues Need Help

View All on GitHub

feat: builtin ROCm support 2 months ago

AI Summary: The task is to add built-in ROCm (Radeon Open Compute) support to the node-llama-cpp project. This involves enabling the use of existing llama.cpp binaries built with ROCm, rather than requiring users to build from source. The current issue is that the build process fails due to missing ROCm libraries, and a solution needs to be implemented to handle this scenario gracefully.

Complexity: 4/5

new feature help wanted

withcatai/node-llama-cpp

1.6K

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level

TypeScript

#ai#bindings#catai#cmake#cmake-js#cuda#embedding#function-calling#gguf#gpu#grammar#json-schema#llama#llama-cpp#llm#metal#nodejs#prebuilt-binaries#self-hosted#vulkan