Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Parse PrimitiveLogicalType::Unknown as Arrow Null DataType #1563

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Sep 5, 2023

This PR adds logic to parse any Parquet fields with Unknown LogicalType as the Arrow Null DataType.

@codecov
Copy link

codecov bot commented Sep 5, 2023

Codecov Report

Patch coverage: 93.75% and project coverage change: -0.01% ⚠️

Comparison is base (fb7b5fe) 83.06% compared to head (417c35c) 83.05%.
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1563      +/-   ##
==========================================
- Coverage   83.06%   83.05%   -0.01%     
==========================================
  Files         391      391              
  Lines       42889    42895       +6     
==========================================
+ Hits        35626    35628       +2     
- Misses       7263     7267       +4     
Files Changed Coverage Δ
src/io/parquet/read/schema/convert.rs 94.60% <80.00%> (-0.09%) ⬇️
src/compute/cast/mod.rs 90.84% <100.00%> (+0.01%) ⬆️

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jaychia
Copy link
Contributor Author

jaychia commented Sep 5, 2023

Also, no unit tests on this yet but would love pointers on how to best test this change.

Should I add a new column in the files generated by parquet_integration/write_parquet.py?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant