thomhurst/Dekaf

AI Summary: This issue identifies an unnecessary array clearing operation in the `BatchArena.ReturnToPool` method. Clearing large byte arrays (typically 1MB+) when returning them to the `ArrayPool<byte>.Shared` consumes CPU time and is redundant as the data is not sensitive and will be overwritten on subsequent use. The proposed fix is to change the `clearArray` parameter to `false`.

AI Summary: The documentation for KafkaConsumer incorrectly states it's not thread-safe, implying all methods must be called from a single thread. The proposed fix clarifies that user-facing API calls are single-threaded, but internal operations utilize background tasks, requiring users to avoid concurrent calls to consumer methods.

AI Summary: The varint encoding logic in the ProtobufSchemaRegistrySerializer is duplicated in both the size calculation and write methods. This duplication poses a maintenance risk, as changes to one might not be reflected in the other, potentially leading to silent data corruption. The proposed fix involves creating a single, shared method for varint encoding.

AI Summary: This issue proposes adding convenience extension methods to the Dependency Injection (DI) builder for registering serializers. Currently, users have to manually construct serializers, but these extensions would simplify the process for common formats like JSON, Avro, and Protobuf, making serializer setup more streamlined.

AI Summary: The `ParseAndSliceResponse` method in `KafkaConnection.cs` currently returns null without providing any diagnostic context when parsing fails. This makes it very difficult to debug issues related to broker protocol violations. The proposed fix involves replacing these silent null returns with descriptive exceptions that include information about the failure point, buffer length, and offset.

AI Summary: This issue proposes adding Brotli compression support to the Dekaf library. Brotli offers improved compression ratios over Gzip and is straightforward to implement using .NET's built-in BrotliStream. The implementation would involve creating a new package and registering Brotli as a custom codec, requiring explicit installation by producers and consumers.

AI Summary: This issue proposes adding an `IPartitionAssignmentStrategy` interface to Dekaf, allowing users to implement custom client-side logic for assigning Kafka partitions to consumers. This would enable advanced scenarios like rack-aware or weighted partition assignment, which are not covered by the existing built-in strategies and offer an alternative to server-side assignment solutions.

AI Summary: This issue identifies performance concerns in `SendSaslMessageAsync` due to the repeated allocation of byte arrays for SASL messages. The suggested fix is to leverage `ArrayPool<byte>` to rent and return these buffers, thereby reducing memory pressure and improving efficiency, especially since these allocations, though infrequent, are avoidable.

AI Summary: This issue identifies a performance bottleneck in Kafka message serialization where a new `SerializationContext` struct is allocated for every key and value serialization. This leads to frequent memory allocations, impacting performance, especially in high-throughput scenarios. The suggested fixes involve reusing `SerializationContext` instances or utilizing stack allocation to mitigate these allocations.