Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rust): casting utf8 to temporal #12072

Merged
merged 3 commits into from
Oct 28, 2023

Conversation

brayanjuls
Copy link
Contributor

Closes #9994 , closes #10478

The implementation I did in this PR support converting strings to date or datetime, it only support specific formats which are the following:

  • %Y-%m-%d
  • %Y-%m-%dT%H:%M:%S
  • %Y-%m-%dT%H:%M:%S.3f
  • %Y-%m-%dT%H:%M:%S%.f%:z

Additionally:

  • I found that timestamp casting for time-unit milliseconds and microseconds is not implemented in arrow2 library, so I opened this issue. This is now implemented in nano-arrow crate inside polars.
  • Converting date times with the following format "2021-01-01 12:54:58.432" or "2021-01-01 12:54:58" is not supported by nano-arrow cast function, it only supports RFC3339(or a variation of it), should the other formats be supported as well ? if so I think we should open a different issue for that.
  • Converting a string number to date is not supported, ie: pl.DataFrame({"x1":["1234"]}).with_columns(**{ "x1-date": pl.col("x1").cast(pl.Date)}), this is why I removed pl.Utf8 from test_from_epoch unit test, should it be supported?

Example

import polars as pl

df = pl.DataFrame({
    "x1": ["2021-01-01"], 
    "x2": ["2021-01-01T12:54:58"],
    "x3": ["2021-01-01T12:54:58.432"],
    "x4": ["2021-01-01T12:54:58.432 +00:00"],
}).with_columns(**{
    "x1-date":pl.col("x1").cast(pl.Date),
    "x2-datetime":pl.col("x2").cast(pl.Datetime),
    "x3-datetime":pl.col("x3").cast(pl.Datetime),
    "x4-datetime-tz":pl.col("x3").cast(pl.Datetime("ms","America/Santiago")),
})

For a full history of this PR please refer to the comments in #10517 which was close due to errors while rebasing.

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature rust Related to Rust Polars labels Oct 27, 2023
Copy link
Collaborator

@MarcoGorelli MarcoGorelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes themselves look good to me, just not sure about the location of the tests

But test_queries seems like quite a generic file, I'm not totally sure what kinds of tests it's meant to be collecting anyway, so maybe that file needs cleaning up regardless of this PR

Over to @stinodego if you have other comments

EDIT: just added a commit to address the above

py-polars/tests/unit/test_queries.py Outdated Show resolved Hide resolved
@ritchie46 ritchie46 merged commit 440e093 into pola-rs:main Oct 28, 2023
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature rust Related to Rust Polars
Projects
None yet
3 participants