Gluten 2025 Roadmap

[FelixYBW]

Description

2024 is almost gone. Time to define 2025 features. Here is some from my mind. Feel free to add more or comment
For more information about the achievements in 2024 Roadmap, please check #4709

Velox backend:

• Spark 4.0 support with minimal JDK 11 support in Gluten 1.5, Deprecate Spark 3.2 in Gluten 1.4 and JDK 8 in Gluten 1.5 ([CORE] Spark 4.0 support #8852)
• Spark 3.3.x/3.4.x/3.5.x upgrade
• Switch to Upstream Velox's official release ([VL] upstream OAP/Velox commits to upstream #8782)
• Stage level resource management(Stage level resource management to handle offheap/onheap memory conflict #4392 )
• Spill enhancement and performance ([VL] Spill related issues tracker #3030 )
• Hash table broadcast support in BHJ (https://docs.google.com/document/d/1upEby9aJnBcMKul7HS5agSr1zwFR63Iq4QL1nYxuFvU/edit?usp=sharing)
• Dictionary support in shuffle ([VL] Dictionary support in shuffle #8855)
• Multile core per task ([VL] Use multiple threads in the same executor #7810 )
• Pyspark Support including Python UDF/Arrow UDF
• Other Accelerator such as GPU, FPGA Support ([Core] Add GPU support in Gluten #8851)
• ARM architecture support
• Stability improvement, full fuzzer support, Result Mismatch issues ([VL] Result mismatch issues tracker #4652 )
• Spark Coverage including Operators and Functions
• Full Datalake support
• File Format support (Json)
• Performance improvement: Spill, Sort, HashAgg, HashJoin, ColumnarShuffle, and others
• Complex type in R2C ([VL] row2VeloxColumnar bad performance #7223)
• Gluten Indicator
• Gluten + Flink + Velox POC (experimental)
• ANSI support
• query trace
• Pushdown scan and filter to remote storage

Community:

• Major Release: 1.4/1.5/1.6/1.7 in Q1/Q2/Q3/Q4 2025
• Minor Release: On Demand
• Apache TLP Submission in Q4 2025

[zhanglistar]

What about supporting Flink framework?

[Myracle]

Any plan or discussion about flink?

[zhztheplayer]

@zhanglistar @Myracle

It waits for discussion.

Does someone want to open / host a discussion for the topic? cc @apache/gluten-committers

[jinchengchenghh]

Do we need to do complex data type support and optimization such as row to column optimization for complex data type?

[FelixYBW]

Do we need to do complex data type support and optimization such as row to column optimization for complex data type?

Yes, currently R2C with complex type is too expensive

[MacChen01]

What about support openEuler OS ?

[FelixYBW]

What about support openEuler OS ?

Can vcpkg=on work on openEuler?

[felipepessoto]

@FelixYBW what you mean by Full Datalake support? Is it to improve support and performance for Delta tables?

[FelixYBW]

@FelixYBW what you mean by Full Datalake support? Is it to improve support and performance for Delta tables?

some features are not supported yet in Velox and Gluten.

[VaibhavFRI]

@FelixYBW what will be included in ARM architecture support? Will Gluten, Velox utilize SVE in ARM to improve performance?

[FelixYBW]

@FelixYBW what will be included in ARM architecture support? Will Gluten, Velox utilize SVE in ARM to improve performance?

It means we will setup CI or jenkins to verify on the ARM platform. SVE is part of the performance optimization work.