Open Issues Need Help
View All on GitHubA minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
AI Summary: The user is experiencing an issue where their application, running in a Docker container using `diffusers` and `litserve`, freezes without error messages when configured to use multiple GPUs (`count: all` in `docker-compose.yml`). While a single GPU setup works correctly, sending requests to the multi-GPU configuration causes the system to hang. The setup involves 2x RTX 3060 12GB cards, with the model occupying approximately 7GB VRAM per card.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
AI Summary: The `LitServer` class incorrectly registers middleware multiple times if initialized with two or more `LitAPI` instances. This happens because the `_register_middleware` method is called within a loop that iterates over the `LitAPI` objects, leading to duplicated middleware entries in `server.app.user_middleware` instead of a single instance.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
AI Summary: The task is to determine how to deploy a Flair-based Named Entity Recognition (NER) model, specifically the 'flair/ner-english-large' model from Hugging Face, using the LitServe framework. This involves understanding how to integrate the Flair model's prediction functionality within the LitServe API structure and potentially handling any specific requirements or challenges posed by the model's size and architecture.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
AI Summary: The task is to fix an import error in the `litserve` Python library caused by the removal of the `_convert_to_content` function from the `mcp` library. This requires investigating the `litserve` codebase to identify where this private function is used and refactor it to use a public API from `mcp` or find an alternative solution. The solution should prevent the use of private APIs to improve the robustness and maintainability of `litserve`.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
AI Summary: The task is to modify the LitServe library to allow users to override the default endpoint paths defined in the OpenAPI specification (`spec`). Currently, the library uses fixed URLs, preventing customization. The solution requires altering the code to accept and utilize user-specified paths for API endpoints.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
AI Summary: The task is to modify the LitServe framework to allow access to request headers within the `request` object passed to the `predict` method of a LitAPI. This is needed to integrate distributed tracing, specifically to extract a parent trace ID from the incoming request headers.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.