Generate thrift definitions using macro from compact-thrift-rs #5909
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
An alternative implementation of a custom thrift decoder, this time based on rust declarative macros instead of a separate code generator.
Fixes #5854, Alternative to #5869
Rationale for this change
The current thrift definitions are not easy to customize and also leave some performance on the table. The macro-based approach here allows renaming fields and replacing types with custom implementations, while still being relatively easy to update.
The downsides of the macro is that error messages in case of typos are not that easy to deciper. Once there are more customizations, updates would need to be done via diff with the official definitions.
Even without customizations like borrowing data, the performance improvement is significant. On my machine, with
native
target and still the default allocator:What changes are included in this PR?
Replaces the thrift structures and custom protocol with a macro that wraps the nearly unmodified
parquet.thrift
definitions.Are there any user-facing changes?
This should mostly be a drop-in replacement, customizations and optimizations could be done in future PRs, for example borrowing data or replacing a
Vec<Encoding>
with a small bitset.One api change is that structs and enum no longer derive the
Eq
trait, onlyPartialEq
.