-
Notifications
You must be signed in to change notification settings - Fork 842
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add check for empty schema in parquet::schema::types::from_thrift_helper
#6990
base: main
Are you sure you want to change the base?
Conversation
|
||
#[test] | ||
// https://github.com/apache/arrow-rs/issues/6988 | ||
fn test_roundtrip_empty_schema() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is @samgqroberts's reproducer from #6988
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @etseidl and @samgqrobert 🙏
I have some suggestions on improving the tests, but I also think we can do that as a follow on PR (or never)
let empty_fields: Vec<Field> = vec![]; | ||
let empty_schema = Arc::new(Schema::new(empty_fields)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very minor: you can use Schema::empty
as well:
let empty_fields: Vec<Field> = vec![]; | |
let empty_schema = Arc::new(Schema::new(empty_fields)); | |
let empty_schema = Arc::new(Schema::new(empty_fields)); |
// read from parquet | ||
let bytes = Bytes::from(parquet_bytes); | ||
let result = ParquetRecordBatchReaderBuilder::try_new(bytes); | ||
result.unwrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should also
- check that the schema is correctly read back
- Verify that the reader works correctly (reading back no batches)
For example, something like:
// read from parquet | |
let bytes = Bytes::from(parquet_bytes); | |
let result = ParquetRecordBatchReaderBuilder::try_new(bytes); | |
result.unwrap(); | |
// read from parquet | |
let bytes = Bytes::from(parquet_bytes); | |
let reader = ParquetRecordBatchReaderBuilder::try_new(bytes).unwrap(); | |
assert_eq!(reader.schema(), &empty_batch.schema()); | |
let batches: Vec<_> = reader.build().unwrap().collect::<ArrowResult<Vec<_>>>().unwrap(); | |
assert_eq!(batches.len(), 0); |
(I ran this locally and it passed so I think we could also make these changes as a follow on PR)
Thanks for the review @alamb. I made the changes you suggested. |
Which issue does this PR close?
Closes #6988.
Rationale for this change
Reading a file with an empty schema will fail in
parquet::schema::types::from_thrift_helper
because the root node in the schema is mistaken for a leaf node.What changes are included in this PR?
Adds a check in
from_thrift_helper
for a root node with no children, and exits early if one is detected.Are there any user-facing changes?
No