lance-format

lance-format/lance

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

80 good first / help-wanted issues · Rust · last activity Jul 14, 2026

6.8K stars 760 forks 6.8K watchers Rust Apache License 2.0

apache-arrow computer-vision data-analysis data-analytics data-centric data-format data-science dataops deep-learning duckdb embeddings llms machine-learning mlops python rust

View on GitHub Website

80 Open Issues Need Help Last updated: Jul 14, 2026

Open Issues Need Help

View All on GitHub

The ProjectionPlan::project_batch function is inefficient and noisy 4 days ago

good first issue Stale

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Remove `protoc` requirement 6 days ago

good first issue Stale

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DOC] Update readme to list the cloud storage supported 19 days ago

documentation good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Implement replacement scan function for Lance Scan in DuckDB 19 days ago

good first issue help wanted duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Offset overflow errors can be confusing for users about 1 month ago

good first issue A-encoding

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feat(namespace): preserve Retry-After metadata on ThrottlingError about 1 month ago

enhancement good first issue feature A-namespace

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Don't materialize any data on `count_rows` if the filter can be satisfied by a scalar index 2 months ago

good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

GPU-enabled index training 2 months ago

good first issue A-python PyTorch

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support StringView type in FTS indices 2 months ago

good first issue A-index

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

refactor: use LazyLock to initialize Field and Schema once 3 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add `slice()` method to `LanceDataset` 3 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Clean up `copy` `rename` logic 4 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

working with pydantic model instances 4 months ago

enhancement good first issue A-python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Expose `io_buffer_size` in Java in `CompactionOptions` 4 months ago

good first issue A-java

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add `n_jobs` parameter to `lance.write_dataset` to speed up writing large in-memory tables 4 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Typecheck `lance/arrow.py` and `tests/test_arrow.py` 4 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make a SVD implementation in Rust and remove the dependency of openblas 4 months ago

enhancement good first issue help wanted rust A-index

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow filtering of projected fields 4 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Zero dimension vectors cause Lance to panic 5 months ago

bug good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

CompactionMetrics doesn't have a repr in Python 5 months ago

good first issue A-python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Display warning if the index is not built for a vector column during query 5 months ago

good first issue help wanted A-python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

implement existing V2 indices as `VectorIndexExtension` 5 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow issue vector search in batch mode 5 months ago

enhancement good first issue help wanted A-python rust A-index

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow dictionary encoding to support more fixed width types 6 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Example converter for nuscenes dataset 6 months ago

good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Support predicate pushdown via pyarrow dataset 6 months ago

good first issue help wanted duckdb benchmark rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support Storage Options in Spark Connector 6 months ago

good first issue feature

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feature: support datafusion expression for row deletions 7 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Parametrize SQL strings 8 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Can't filter for vector indices in `describe_indices` 8 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add `ReadFileError` and `DecodeColumnError` wrappers to provide better context for decoding errors 8 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Write specification doc describing schema format 9 months ago

documentation good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Column Statistic improvment list after MVP 9 months ago

enhancement help wanted feature file-storage format

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add general compression for large block arrays 9 months ago

AI Summary: The issue proposes adding general compression capabilities for large block arrays, specifically targeting large dictionary arrays. Currently, block compression is limited to bitpacking, and this enhancement aims to improve support and efficiency for data types that are not well-suited for bitpacking.

Complexity: 3/5

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add optimizer rule to change `LanceScanExec -> FilterExec` into `FilteredReadExec` 9 months ago

good first issue performance

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Remove duplicate datafusion code 10 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Enhance not found errors with suggestions 10 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Review use of `Error::io()` to find cases where that error should be a different type 10 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

opening non-existent dataset should return `Error::NotFound` instead of `Error::IO` 11 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

chore: consolidate Dataframe ProjectExec and lance ProjectExec 11 months ago

good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Tracking Lance Datasets (DNC) 11 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

enable code coverage for python tests 11 months ago

enhancement good first issue help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

`LanceFragment.to_batches` not respecting `filter` kwarg 11 months ago

bug good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow relative paths in `IvfModel.save()` and `PqModel.save()` 11 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

All binary array tests generate fixed size binary arrays 11 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Exclude sign bit from bitpacked encoding if all values are negative 11 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Fragment::create_from_file not supported with v2 files 11 months ago

help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make it possible to pass `storage_options` to `LanceFileReader` and `LanceFileWriter` 11 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

lance.torch.data.LanceDataset(to_tensor_fn=...) typehint inconsistent with usage. 11 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

pylance write to local disk with a path started with `s3://` 11 months ago

bug good first issue help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Support export table from DuckDB to lance 11 months ago

enhancement help wanted duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Create pylance build for conda environment 11 months ago

good first issue help wanted python ci

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Expose vector search capability via duckdb extension 11 months ago

good first issue help wanted duckdb rust vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Filters pushdown via native lance reader (`lance_scan`) 11 months ago

enhancement good first issue duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support version metadata when writing a dataset 11 months ago

enhancement help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[Duckdb] Zero copy Arrow Array/RecordBatch to duckdb Vector/DataChunk conversion 11 months ago

help wanted duckdb arrow rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support Arrow Tensor type 11 months ago

enhancement help wanted python arrow rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Python: fix broken fragment APIs 11 months ago

good first issue python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make a matrix multiplication implementation in Rust 11 months ago

enhancement good first issue help wanted python rust vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Provide better guidance for users creating indices 11 months ago

documentation good first issue help wanted python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

IVF_SQ index 11 months ago

enhancement help wanted rust vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support `regexp_match` for `LargeUtf8` 11 months ago

good first issue help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Test which Arrow and Pandas datatypes are supported in write APIs 11 months ago

good first issue help wanted python ci arrow

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow users to set a distance threshold to consider two vectors a "very similiar" 11 months ago

enhancement good first issue help wanted python vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Index out of bounds using lance extension in duckdb v0.7.0 11 months ago

bug good first issue python duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make PyO3 functions as async function 11 months ago

enhancement good first issue python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make numpy optional pylance dependency 11 months ago

enhancement good first issue python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Merge with more than a single column for `left_on` possible? 11 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

added a `localstore` scheme in `object_store.rs` for testing 11 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Enable stricter clippy lints 11 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feat: provide a Java SDK to read / write lance format 11 months ago

enhancement help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Fix torch related tests 11 months ago

bug good first issue help wanted python PyTorch

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

`load_indices` should include index config 11 months ago

enhancement good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

lance run benchmark panic 11 months ago

bug good first issue help wanted benchmark rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Android build fails due to `aarch64` in Rust 11 months ago

good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

V2 manifest path in Java 11 months ago

good first issue java

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Does lance support hdfs access? 11 months ago

enhancement help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Consolidate the logic between ZoneMap and BloomFilters indexes 11 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

The `InstrumentedRecordBatchStreamAdapter` is often misused and double counts CPU time 11 months ago

bug good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feat: bytes read and bytes write mectirc for per lance scanner and writer 11 months ago

help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust