Open Issues Need Help
View All on GitHubAI Summary: Users are encountering a `ValueError` when converting a Spark DataFrame, created with `spark.range()` and `withColumns()`, to an Apache Arrow table using `df.toArrow()`. The error, "Target schema's field names are not matching the table's field names," occurs sporadically (approximately 50% of the time), making it difficult to debug. A minimal reproducible example is provided.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
AI Summary: Investigate and replicate Spark's type casting logic for null functions in the LakeSail Sail project, ensuring parity with Spark's behavior. This involves analyzing Spark's source code and potentially implementing the logic as user-defined functions (UDFs) within Sail, also considering the broader applicability of this casting behavior to other scenarios.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.
AI Summary: Implement support for five new Spark 4.0 functions (`>>`, `<<`, `nullifzero`, `zeroifnull`, `dayname`) within the LakeSail Sail framework, ensuring compatibility with its existing Spark SQL and DataFrame API.
LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.