Eventual-Inc/Daft

string expressions parity with pyspark about 9 hours ago

good first issue help wanted p2 (backlog)

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

functions decorated with `daft.func` don't respect the input type signature 3 days ago

enhancement help wanted p2 (backlog) types

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

List-return UDF exceptions are swallowed as "Need at least 1 series to perform concat" 5 days ago

bug help wanted p1 fix

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support upsert functionality in write_iceberg 6 days ago

enhancement good first issue data-catalogs iceberg p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

math expressions parity with pyspark 6 days ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`with_column`/`with_columns` improvements 7 days ago

enhancement help wanted p2 (backlog) python refactor accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: Add `to_map` constructor function 8 days ago

enhancement good first issue expression p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

perf(optimizer): eliminate sorts preceded by another sort 13 days ago

enhancement help wanted p2 (backlog) perf

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Implement sample by size on ray runner 21 days ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: Add top-level `daft.concat()` for concatenating multiple DataFrames 23 days ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

chore: Bump minimum PyArrow version to 16 about 2 months ago

enhancement good first issue accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Parameterized queries to prevent SQL injection attacks about 2 months ago

enhancement help wanted p2 (backlog) sql

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Integration test suites for catalog implementations about 2 months ago

enhancement help wanted chore data-catalogs p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Fix round-trip write/read issues from Parquet/CSV/JSON 2 months ago

bug good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: List flatten: List[List[T]] -> List[T] 2 months ago

enhancement good first issue

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat(dashboard): render Repartition spec in human-readable form instead of debug repr 2 months ago

enhancement good first issue p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`read_text` and `read_blob` functions 3 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for Arrow PyCapsule Interface 3 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Optimizer: common subplan elimination 3 months ago

help wanted p2 (backlog) perf accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

support writing csv without headers 3 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Remove calling functions to get Python DataTypes 3 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

fix: S3 HTTP client ignores HTTP_PROXY/HTTPS_PROXY environment variables 3 months ago

enhancement help wanted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

temporal expressions parity with pyspark 3 months ago

good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`expr.list.map(...)` and `expr.list.filter(...)` 3 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[FEAT] User-defined aggregate functions 3 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Reading Daft-created partitioned Parquet with pandas fails 3 months ago

bug help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add support for `cast(DataType.duration(s/ms/us/ns)` 3 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

better error reporting for invalid sql statements 3 months ago

enhancement help wanted p2 (backlog) sql

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support point-in-time-join on a given set of dataframes 3 months ago

enhancement help wanted p1 accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Reduce total binary size by rewriting code 3 months ago

help wanted p1 accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

bug: coalesce does not exit early when it encounters a non-null value 3 months ago

bug help wanted p2 (backlog) perf accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

add `read_arrow` and `read_arrow_stream` methods 4 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for rolling joins and other special joins 4 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Missing support for Float16 type `FixedShapeTensorArray` or `ArrowTensorArray` from parquet/pyarrow sources 4 months ago

bug help wanted p2 (backlog) types accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add fp16 type support 4 months ago

enhancement help wanted p2 (backlog) types accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

fix the dev release always showing `0.30.0-dev` 4 months ago

bug help wanted build p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[PERF] Eliminate redundant HEAD requests when reading parquet 4 months ago

help wanted p2 (backlog) perf

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[SQL] Add SAMPLE LogicalPlan support 4 months ago

help wanted p2 (backlog) sql accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support selector expressions 4 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Window function support 4 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

flatten nested error messages/better error handling 4 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`df.show` max_width does not work without manually setting the `format` option as well 4 months ago

bug good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add Kafka Data Source Support for Streaming Data Processing 4 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

add support for extended glob features, such as numeric ranges `{0..100}` 4 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support various image hash functions for deduplication. 4 months ago

enhancement good first issue help wanted p2 (backlog) types

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support version-hint.txt for reading iceberg tables 4 months ago

enhancement good first issue iceberg p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Terminal rendering of map types displays internal key-value struct 5 months ago

bug good first issue p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`ddof` argument to `stddev` 5 months ago

enhancement good first issue help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add median function 5 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`daft.read_deltalake` returns a table when an invalid value for `version` is provided 5 months ago

bug good first issue data-catalogs p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support Dataframe cache/persist 5 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Assert Dataframe and schema for testing 5 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support `.as_T` cast methods where `T` is a Daft type with possible parameters 5 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add a `.list.contains()` expression 5 months ago

enhancement good first issue expression p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

time zone conversion features 5 months ago

enhancement good first issue help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Allowing GroupedDataFrame.agg_concat to also take in a delimiter like Expression.list.join 5 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Expression.var 5 months ago

AI Summary: This is a feature request to add `var()` (variance) support for expressions within Narwhals, specifically targeting Daft columns like `daft.col('a').var()`. The request also suggests including a `ddof` argument for controlling the degrees of freedom in the variance calculation.

Complexity: 3/5

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support specifying embedding dimensions for all providers 5 months ago

AI Summary: This feature request seeks to extend the `dimensions` parameter for text embeddings to all supported providers, as it currently only works for OpenAI. The goal is to implement dimension specification for other providers where feasible (e.g., using `truncate_dim` for `SentenceTransformer`) and to fail gracefully if a model does not support this functionality.

Complexity: 3/5

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

read_csv support `scan_tasks_min_size_bytes ` and `scan_tasks_max_size_bytes` to prevent single file too big 5 months ago

enhancement good first issue needs triage

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

custom quote characters when writing csv 5 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add support for Series[start:end] 5 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

custom delimiter when writing csv 5 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Explode returns null on empty or NULL lists 5 months ago

bug good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

custom date & timestamp formatting when writing csv 6 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`uuid` function 6 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Allow overriding embedding dimension precision in OpenAI compatible embedding 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

daft emits DeprecationWarning on import 7 months ago

bug help wanted p1

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add support for writing `_SUCCESS` files in parquet operations, similar to Spark 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Make casting image to tensor easier 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

string casing functions 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[EXPRESSIONS]: Distance functions 7 months ago

enhancement good first issue expression p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Hash rows of dataframe 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Missing mypy_boto3_glue in optional dependencies 7 months ago

bug good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

make it easier to enable different logging levels 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

chore: remove `HasMoreOutput(None)` from `StreamingSinkOutput` 7 months ago

enhancement good first issue p2 (backlog) needs triage

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add examples for using sessions with providers and embed_text 7 months ago

documentation enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Daft doesn't work with OneLake 7 months ago

bug help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Expression.pow 7 months ago

enhancement good first issue help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Refactor the read/write lance API based on the Lance Rust SDK 7 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for `SHOW TABLES LIKE ...`for MemoryCatalog 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Apache Hudi Support Roadmap 7 months ago

enhancement help wanted p2 (backlog) feat

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

enh: DataFrame.join_asof 7 months ago

enhancement help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Merge support for DeltaLake 7 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support UUID data type 7 months ago

enhancement help wanted p2 (backlog) types

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Supporting fractional cpu count on the `ray` runner 7 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Comparison operators for list and struct types 7 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for uint256/int256 7 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Allow adding indices to explodes 7 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

alternative display options for plans 7 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add better docstrings for reading from s3 7 months ago

documentation good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

ability to write to single parquet file 7 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[Catalogs] [Delta Lake] Add support for reading tables with column mappings 7 months ago

enhancement help wanted data-catalogs p2 (backlog) delta-lake

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: Add text embedding dimension override for OAI provider 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[Catalogs] [Delta Lake] Add support for reading tables with deletion vectors 8 months ago

help wanted data-catalogs p2 (backlog) delta-lake

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

improve df display of File[Video] 8 months ago

enhancement good first issue

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support simple case and searched cased with list of branches 9 months ago

AI Summary: This feature request proposes enhancing the `case` statement DSL to mirror SQL's simple and searched case expressions. The aim is to improve programmatic construction, type safety, and reduce ambiguity by allowing branches (conditions and values) to be passed as a list of `when` objects, rather than individual expressions. This new API would eliminate the need for custom helpers and provide a more consistent and type-safe way to build conditional logic.

Complexity: 4/5

enhancement good first issue help wanted p1

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

mypy error for daft.func 9 months ago

bug help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support `avg` as an alias for `mean` in dataframes 10 months ago

enhancement good first issue

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Compatibility with PyIceberg v0.10 10 months ago

enhancement help wanted iceberg p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support specifying different Conda Env for different UDFs 10 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Open Issues Need Help