Lightning-AI/litData

AI Summary: The task requires fixing a bug in the LitData library's streaming cache. The bug involves stale lock files (e.g., *.zstd.bin.lock) being left behind after compressed data chunks are deleted from the cache. The solution involves modifying the cache cleanup process to ensure that all associated lock files (both compressed and uncompressed) are removed when a chunk is deleted.

AI Summary: Implement a `shuffle` parameter in the `train_test_split` function of the LitData library. This parameter should control whether the dataset is shuffled before splitting into training, validation, and testing sets. The default behavior should remain to shuffle the data, but the new parameter should allow users to disable shuffling if needed, for example, to maintain the original order of the data.

AI Summary: Update the README file's "Get started" badge link to a working URL. The current link is broken and needs to be replaced with a valid and relevant link from the Lightning AI documentation, such as https://lightning.ai/docs/overview/optimize-data/optimize-datasets.

AI Summary: The task requires modifying the `ParallelStreamingDataset` class in the LitData library. The current implementation doesn't correctly resume from a previous epoch if the iteration is broken early. The solution involves adjusting the state loading and saving mechanism within the `iter()` method, potentially separating the cycling and resuming functionalities by adding a `resume` option to the class. Unit tests should be added to ensure the fix works as intended.

AI Summary: Optimize the `train_test_split` function in the LitData library to improve its performance when dealing with a large number of data chunks. The current implementation has O(N^2) complexity due to filename matching during subsampling; this needs to be reduced to O(N) or better, likely by using sets for efficient membership checking.

AI Summary: The task is to debug a performance issue in the LitData `StreamingDataLoader`. A noticeable delay occurs between epochs when training a model, even with a local dataset. The solution requires investigating the `StreamingDataLoader`'s behavior, potentially focusing on aspects like data prefetching, buffer management, or inter-process communication overhead. Reproducing the issue using the provided code and identifying the root cause are key steps.

AI Summary: Debug and fix a bug in the LitData library where iterating over a Hugging Face dataset with `num_workers > 0` after a previous iteration with `num_workers = 0` causes the script to hang. The solution likely involves investigating resource management and potential conflicts between the different worker configurations within the `StreamingDataLoader` when handling Hugging Face datasets.