Add simdutf8 feature to make simdutf8 optional, consolidate check_valid_utf8
#6979
+81
−28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
commented
Jan 14, 2025
| - `lz4` (default) - support for parquet using `lz4` compression | ||
| - `zstd` (default) - support for parquet using `zstd` compression | ||
| - `snap` (default) - support for parquet using `snappy` compression | ||
| - `arrow` (default) - support for reading / writing [`arrow`] arrays to / from Parquet |
There was a problem hiding this comment.
this was a drive by cleanup b/c I can't help myself now that @etseidl pointed out the capitalization inconsistency with Parquet and parquet
| - `cli` - parquet [CLI tools](https://github.com/apache/arrow-rs/tree/main/parquet/src/bin) | ||
| - `crc` - enables functionality to automatically verify checksums of each page (if present) when decoding | ||
| - `experimental` - Experimental APIs which may change, even between minor releases | ||
| - `simdutf8` (default) - Use the [`simdutf8`] crate for SIMD-accelerated UTF-8 validation |
There was a problem hiding this comment.
this is the new feature
| /// | ||
| /// [`simdutf8`]: https://crates.io/crates/simdutf8 | ||
| #[inline(always)] | ||
| pub fn check_valid_utf8(val: &[u8]) -> Result<()> { |
There was a problem hiding this comment.
I implemented @etseidl 's suggestion #6668 (comment)
for encapsulation to make the code / use eaiser to understand
| @@ -131,6 +131,9 @@ pub mod data_type; | |||
| pub use self::encodings::{decoding, encoding}; | |||
|
|
|||
| experimental!(#[macro_use] mod util); | |||
|
|
|||
| pub use util::utf8; | |||
There was a problem hiding this comment.
This publically exports the utf8 module as part of the parquet modules, which is needed for adding a doc test.
I was thinking this might be useful for other users (to use the same utf8 validation library) but I can also be convinced to avoid adding this function to the public API of the
alamb
force-pushed
the
alamb/simdutf8-optional
branch
from
7d24c25 to
f038faf
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
simdutf8#6668Rationale for this change
Per @Dandandan's comments on #6668 (comment), let's make this package optional to provide an "escape hatch" if anyone downstream hits issues.
I made it a separate PR per @doki23's suggestion #6668 (comment)
What changes are included in this PR?
simdutf8feature, which controls the use ofsimdutf8for utf8 validationAre there any user-facing changes?