Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Improve literals for temporal subclasses #18998

Merged
merged 1 commit into from
Sep 29, 2024

Conversation

mcrumiller
Copy link
Contributor

@mcrumiller mcrumiller commented Sep 28, 2024

Fixes #18906.

import pandas as pd
import polars as pl

pl.select(a=pl.lit(pd.Timedelta(minutes=1)))
# shape: (1, 1)
# ┌──────────────┐
# │ a            │
# │ ---          │
# │ duration[μs] │
# ╞══════════════╡
# │ 1m           │
# └──────────────┘

This update pushes allow_object into the AnyValue conversion, and by doing so avoids the double-gil lock for subclassed items (@ritchie46 I think it's what you were thinking in this comment). I think it should help tidy up some of the subclass issues with temporals. I also added checks on time and timedelta temporal classes as well, as is the case in the linked issue where pd.Timedelta subclasses datetime.timedelta.

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Sep 28, 2024
} else if ob.is_instance_of::<PyDict>() {
get_struct
} else if ob.hasattr(intern!(py, "_s")).unwrap() {
get_list_from_series
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was unused as far as I can tell. Removing it didn't fail any tests. The path to a literal list is elsewhere.

Copy link

codecov bot commented Sep 28, 2024

Codecov Report

Attention: Patch coverage is 98.41270% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.88%. Comparing base (901b243) to head (a61f379).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-python/src/conversion/any_value.rs 97.43% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #18998   +/-   ##
=======================================
  Coverage   79.88%   79.88%           
=======================================
  Files        1524     1524           
  Lines      207634   207643    +9     
  Branches     2904     2904           
=======================================
+ Hits       165873   165882    +9     
  Misses      41214    41214           
  Partials      547      547           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mcrumiller mcrumiller marked this pull request as draft September 28, 2024 21:06
@mcrumiller
Copy link
Contributor Author

mcrumiller commented Sep 28, 2024

I'm unable to replicate the failing test, even after installing same Python (3.12.16). @MarcoGorelli any ideas?

Edit: rebased and it went away.

@mcrumiller mcrumiller marked this pull request as ready for review September 29, 2024 02:02
@ritchie46
Copy link
Member

Thanks @mcrumiller. I think this is a good improvement as we now convert in one location.

@ritchie46 ritchie46 merged commit ebd5302 into pola-rs:main Sep 29, 2024
25 checks passed
@mcrumiller mcrumiller deleted the temporal-lit branch September 29, 2024 12:48
@ritchie46
Copy link
Member

@mcrumiller can you take a look. This test failed in the ci: https://github.com/pola-rs/polars/actions/runs/11093722576/job/30820360138?pr=18930

@mcrumiller
Copy link
Contributor Author

Yikes, sorry. Yeah let me look.

@mcrumiller
Copy link
Contributor Author

Must be something about installed timezone modules, I can't reproduce. I'll work on the setup to figure out what's going on.

@mcrumiller
Copy link
Contributor Author

For some reason get_datetime is being called sometimes when a subclass of time and timedelta are used, resulting in an invalid access to the tzinfo property. @MarcoGorelli do you have any clue what could trigger this?

@mcrumiller
Copy link
Contributor Author

New theory: same class name defined in same function in multiple threads are colliding, even with different inheritence definitions.

@mcrumiller
Copy link
Contributor Author

@ritchie46 I've added PR #19012 which I think resolves the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot create literal for pd.Timedelta with allow_object=False
2 participants