lightbench: a lightweight tool for benchmarking LLMs 🛠️

benchmark llm llm-inference
2 Open Issues Need Help Last updated: Sep 12, 2025

Open Issues Need Help

View All on GitHub
Add Joule result about 1 month ago
good first issue wontfix

lightbench: a lightweight tool for benchmarking LLMs 🛠️

Python
#benchmark#llm#llm-inference

AI Summary: The task is to modify the `ClaudeLoader` in the `lightbench` project to correctly handle system messages when interacting with the Anthropic API. This involves parsing the input prompt to separate the system message from the user prompt and then passing them as separate arguments to the Anthropic API's `messages.create` function.

Complexity: 3/5
help wanted good first issue

lightbench: a lightweight tool for benchmarking LLMs 🛠️

Python
#benchmark#llm#llm-inference