All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Breaking: Using DataFusion's
enable_ident_normalization = false
setting to work with upper case identifiers without needing to put quotes everywhere - Updated Rust toolchain
- Updated to
datafusion v41
- Exposed JSON extensions from
datafusion-functions-json
library
- Building
arm64
version of the image to support Apple M-series silicon without QEMU overhead
- Reverted to use
int32
andint64
forop
andoffset
respectively to preserve compatibility with Spark engine until Spark's Parquet version is upgraded.
- Migration to ODF changelog schema
- Upgraded to new ODF engine protocol
- Upgraded to
datafusion v34
- Upgraded to new ODF schemas
- Upgraded to
datafusion v30.0.0
- Enabled compression and using better encoding for Parquet files
- All timestamps will be coerced into
Timestamp(MILLIS,true)
for better Flink compatibility
- Watermark computation bug
- Upgraded to latest
datafusion
(fixes several known issues)
- Improved error handling
- Using patched
arrow
/parquet
version that readsTIMESTAMP_MILLIS
type as UTC.
- Writing
offset
asInt64
type instead ofUInt64
for compatibility with other engines.
- Initial engine prototype
- Keeping a CHANGELOG