Eventual-Inc

Eventual-Inc/Daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

100 good first / help-wanted issues · Rust · last activity Jul 1, 2026

5.6K stars 504 forks 5.6K watchers Rust Apache License 2.0
ai-engineering ai-pipeline arrow artificial-intelligence big-data data-engineering distributed distributed-computing distributed-systems embeddings etl huggingface iceberg machine-learning multimodal parquet python ray rust
View on GitHub Website
100 Open Issues Need Help Last updated: Jul 1, 2026

Open Issues Need Help

View All on GitHub
[PERF] Eliminate redundant HEAD requests when reading parquet 15 days ago
help wanted p2 (backlog) perf
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support `CROSS JOIN` in SQL 17 days ago
enhancement good first issue help wanted p2 (backlog) sql
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
List-return UDF exceptions are swallowed as "Need at least 1 series to perform concat" 20 days ago
bug help wanted p1 fix
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support ANSI SQL `SUBSTRING(expr [FROM start] [FOR len])` syntax 20 days ago
enhancement good first issue help wanted p2 (backlog) sql
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Writer for WebDatasets File Format 26 days ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support using OpenDAL Python library for additional filesystems about 1 month ago
enhancement help wanted perf
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Python Interface for IO extensions about 1 month ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Reader for WebDatasets File Format about 1 month ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support DataSources that wrap DataFrame templates about 2 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support ADBC reads and writes from databases about 2 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
perf(optimizer): eliminate sorts preceded by another sort 2 months ago
enhancement help wanted p2 (backlog) perf
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add split_part string function 3 months ago
enhancement good first issue expression p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
temporal expressions parity with pyspark 3 months ago
good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Truncate IOConfig prints in plan outputs (`df.explain()`) 3 months ago
enhancement help wanted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`read_text` and `read_blob` functions 3 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Migrate HudiScanOperator to DataSource 3 months ago
good first issue help wanted chore p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Migrate IcebergScanOperator to DataSource 3 months ago
good first issue help wanted chore p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Migrate DeltaLakeScanOperator to a DataSource 3 months ago
good first issue help wanted chore p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Mirror load_datasets api for read_huggingface 3 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Window function support 3 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
string expressions parity with pyspark 3 months ago
good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
functions decorated with `daft.func` don't respect the input type signature 4 months ago
enhancement help wanted p2 (backlog) types
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support upsert functionality in write_iceberg 4 months ago
enhancement good first issue data-catalogs iceberg p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
math expressions parity with pyspark 4 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`with_column`/`with_columns` improvements 4 months ago
enhancement help wanted p2 (backlog) python refactor accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
feat: Add `to_map` constructor function 4 months ago
enhancement good first issue expression p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Implement sample by size on ray runner 4 months ago
enhancement help wanted p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
feat: Add top-level `daft.concat()` for concatenating multiple DataFrames 4 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
chore: Bump minimum PyArrow version to 16 5 months ago
enhancement good first issue accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Parameterized queries to prevent SQL injection attacks 5 months ago
enhancement help wanted p2 (backlog) sql
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Integration test suites for catalog implementations 6 months ago
enhancement help wanted chore data-catalogs p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Fix round-trip write/read issues from Parquet/CSV/JSON 6 months ago
bug good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
feat: List flatten: List[List[T]] -> List[T] 6 months ago
enhancement good first issue
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
feat(dashboard): render Repartition spec in human-readable form instead of debug repr 6 months ago
enhancement good first issue p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support for Arrow PyCapsule Interface 6 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Optimizer: common subplan elimination 6 months ago
help wanted p2 (backlog) perf accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
support writing csv without headers 6 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Remove calling functions to get Python DataTypes 6 months ago
enhancement help wanted p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
fix: S3 HTTP client ignores HTTP_PROXY/HTTPS_PROXY environment variables 7 months ago
enhancement help wanted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`expr.list.map(...)` and `expr.list.filter(...)` 7 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
[FEAT] User-defined aggregate functions 7 months ago
enhancement help wanted p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Reading Daft-created partitioned Parquet with pandas fails 7 months ago
bug help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add support for `cast(DataType.duration(s/ms/us/ns)` 7 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
better error reporting for invalid sql statements 7 months ago
enhancement help wanted p2 (backlog) sql
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support point-in-time-join on a given set of dataframes 7 months ago
enhancement help wanted p1 accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Reduce total binary size by rewriting code 7 months ago
help wanted p1 accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
bug: coalesce does not exit early when it encounters a non-null value 7 months ago
bug help wanted p2 (backlog) perf accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
add `read_arrow` and `read_arrow_stream` methods 7 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support for rolling joins and other special joins 7 months ago
enhancement help wanted p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Missing support for Float16 type `FixedShapeTensorArray` or `ArrowTensorArray` from parquet/pyarrow sources 7 months ago
bug help wanted p2 (backlog) types accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add fp16 type support 7 months ago
enhancement help wanted p2 (backlog) types accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
fix the dev release always showing `0.30.0-dev` 7 months ago
bug help wanted build p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
[SQL] Add SAMPLE LogicalPlan support 7 months ago
help wanted p2 (backlog) sql accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support selector expressions 7 months ago
enhancement help wanted p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
flatten nested error messages/better error handling 7 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`df.show` max_width does not work without manually setting the `format` option as well 8 months ago
bug good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add Kafka Data Source Support for Streaming Data Processing 8 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
add support for extended glob features, such as numeric ranges `{0..100}` 8 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support various image hash functions for deduplication. 8 months ago
enhancement good first issue help wanted p2 (backlog) types
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support version-hint.txt for reading iceberg tables 8 months ago
enhancement good first issue iceberg p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Terminal rendering of map types displays internal key-value struct 8 months ago
bug good first issue p2 (backlog) accepted
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`ddof` argument to `stddev` 9 months ago
enhancement good first issue help wanted p2 (backlog) narwhals
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add median function 9 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`daft.read_deltalake` returns a table when an invalid value for `version` is provided 9 months ago
bug good first issue data-catalogs p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support Dataframe cache/persist 9 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Assert Dataframe and schema for testing 9 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support `.as_T` cast methods where `T` is a Daft type with possible parameters 9 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add a `.list.contains()` expression 9 months ago
enhancement good first issue expression p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
time zone conversion features 9 months ago
enhancement good first issue help wanted p2 (backlog) narwhals
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Allowing GroupedDataFrame.agg_concat to also take in a delimiter like Expression.list.join 9 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Expression.var 9 months ago

AI Summary: This is a feature request to add `var()` (variance) support for expressions within Narwhals, specifically targeting Daft columns like `daft.col('a').var()`. The request also suggests including a `ddof` argument for controlling the degrees of freedom in the variance calculation.

Complexity: 3/5
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support specifying embedding dimensions for all providers 9 months ago

AI Summary: This feature request seeks to extend the `dimensions` parameter for text embeddings to all supported providers, as it currently only works for OpenAI. The goal is to implement dimension specification for other providers where feasible (e.g., using `truncate_dim` for `SentenceTransformer`) and to fail gracefully if a model does not support this functionality.

Complexity: 3/5
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
read_csv support `scan_tasks_min_size_bytes ` and `scan_tasks_max_size_bytes` to prevent single file too big 9 months ago
enhancement good first issue needs triage
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
custom quote characters when writing csv 9 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add support for Series[start:end] 9 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
custom delimiter when writing csv 9 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Explode returns null on empty or NULL lists 9 months ago
bug good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
custom date & timestamp formatting when writing csv 9 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
`uuid` function 10 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Allow overriding embedding dimension precision in OpenAI compatible embedding 10 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
daft emits DeprecationWarning on import 10 months ago
bug help wanted p1
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add support for writing `_SUCCESS` files in parquet operations, similar to Spark 10 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Make casting image to tensor easier 11 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
string casing functions 11 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
[EXPRESSIONS]: Distance functions 11 months ago
enhancement good first issue expression p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Hash rows of dataframe 11 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Missing mypy_boto3_glue in optional dependencies 11 months ago
bug good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
make it easier to enable different logging levels 11 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
chore: remove `HasMoreOutput(None)` from `StreamingSinkOutput` 11 months ago
enhancement good first issue p2 (backlog) needs triage
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Add examples for using sessions with providers and embed_text 11 months ago
documentation enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Daft doesn't work with OneLake 11 months ago
bug help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Expression.pow 11 months ago
enhancement good first issue help wanted p2 (backlog) narwhals
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Refactor the read/write lance API based on the Lance Rust SDK 11 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support for `SHOW TABLES LIKE ...`for MemoryCatalog 11 months ago
enhancement good first issue p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Apache Hudi Support Roadmap 11 months ago
enhancement help wanted p2 (backlog) feat
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
enh: DataFrame.join_asof 11 months ago
enhancement help wanted p2 (backlog) narwhals
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Merge support for DeltaLake 11 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Support UUID data type 11 months ago
enhancement help wanted p2 (backlog) types
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Supporting fractional cpu count on the `ray` runner 11 months ago
enhancement help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust
Comparison operators for list and struct types 11 months ago
enhancement good first issue help wanted p2 (backlog)
Eventual-Inc/Daft
5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust
#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust