Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(datasets): Improved compatibility, functionality and testing for SnowflakeTableDataset #881

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open

feat(datasets): Improved compatibility, functionality and testing for SnowflakeTableDataset #881

wants to merge 40 commits into from
+259 −230

Conversation

tdhooghe
Copy link

@tdhooghe tdhooghe commented Oct 10, 2024 •
edited
Loading

Description

  • Enable saving a Pandas DataFrame directly into a SnowflakeTable, i.e. to ingest a .csv directly into Snowflake within Kedro
  • Update tests to use Snowpark's local testing framework
  • Updated dependencies

Development notes

Pytest up until 3.11 (Snowpark unfortunately not compatible with 3.12 yet)
See https://github.com/kedro-org/kedro-plugins/actions/runs/11443609488
image

Bump of cloudpickle required to allow for snowflake-snowpark-python >= 1.23
image

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes

@tdhooghe tdhooghe changed the title (feature)Datasets: save pandas df directly to SnowflakeTable feat(datasets): save pandas df directly to SnowflakeTable Oct 10, 2024
@tdhooghe tdhooghe mentioned this pull request Oct 10, 2024
Merged
4 tasks
@tdhooghe tdhooghe force-pushed the feature/save-pd-to-snowflaketable branch 2 times, most recently from 0e281c2 to c9c1872 Compare October 10, 2024 19:47
@tdhooghe tdhooghe mentioned this pull request Oct 15, 2024
Open
@tdhooghe tdhooghe force-pushed the feature/save-pd-to-snowflaketable branch from dc88a84 to 0a14bb7 Compare October 21, 2024 14:39
tdhooghe and others added 23 commits October 21, 2024 17:22
@tdhooghe
check and convert pd df to snowflake df
2305c3d 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add tests
07ef67b 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert import to previous state
04928ba 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
relax snowpark version
77551d6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert pyproject.toml
fbb3374 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
bump snowpark version
fb621ae 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
relax python version for snowpark
cb320bd 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
downgrade snowpark to 1.22
628def6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
upgrade cloudpickle to 2.2.1
136fb09 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
feat(datasets): create separate ibis.FileDataset (kedro-org#842)
fe3bfbe 
* feat(datasets): create separate `ibis.FileDataset`

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): deprecate `TableDataset` file I/O

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets): implement `FileDataset` versioning

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): try `os.path.exists`, for Windows

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* revert(datasets): use pathlib, ignore Windows test

Refs: b7ff0c7

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* docs(datasets): add `ibis.FileDataset` to contents

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets)!: expose `load` and `save` publicly

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): remove second filepath assignment

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@ankatiyar @tdhooghe
ci(docker): Update error code in e2e test (kedro-org#888)
fce2974 
Update error code in e2e test

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
update docstring
60b4a86 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
test ci/cd without python version 3.12
3e578c6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
fix linting error
a161191 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add max python version for compatibility
e4509bc 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
fix error
c98aef8 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@merelcht @deepyaman @tdhooghe
chore: Update PR template with checkbox for core dataset contribution (…
e4f1c3c 
…kedro-org#891)

* Update PR template with checkbox for core dataset contribution

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

* Update .github/PULL_REQUEST_TEMPLATE.md

Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Fix lint

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

---------

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@ankatiyar @tdhooghe
ci(datasets): Fix mypy errors (kedro-org#893)
bcec735 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
fix(datasets): default to DuckDB in in-memory mode (kedro-org#897)
c243c6e 
* fix(datasets): default to DuckDB in in-memory mode

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* test(datasets): use `object()` sentinel as default

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* docs(datasets): add default database to RELEASE.md

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
build(datasets): make Kedro-Datasets 5.1.0 release (kedro-org#887)
388f455 
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@merelcht @tdhooghe
ci: Add GH action to check for TSC votes on core dataset changes (ked…
3c85b1f 
…ro-org#896)

* Add GH action to check for TSC votes on core dataset changes
* Ignore TSC vote action in gatekeeper
* Trigger TSC vote action only on changes in core dataset

---------

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
update testing framework and docstrings
101696c 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
Update kedro-datasets/kedro_datasets/snowflake/snowpark_dataset.py
c1eeb77 
Signed-off-by: Thomas <thomas.dhooghe95@gmail.com>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
validate and check table name for nones
b9bf73e 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
remove breakpoint
2c804c9 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe tdhooghe force-pushed the feature/save-pd-to-snowflaketable branch from cee8467 to 2c804c9 Compare October 21, 2024 15:22
tdhooghe and others added 6 commits October 21, 2024 18:05
@tdhooghe
change doc type hinting for tests to pass
7b7811d 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
feat(datasets): create separate ibis.FileDataset (kedro-org#842)
23b056a 
* feat(datasets): create separate `ibis.FileDataset`

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): deprecate `TableDataset` file I/O

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets): implement `FileDataset` versioning

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): try `os.path.exists`, for Windows

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* revert(datasets): use pathlib, ignore Windows test

Refs: b7ff0c7

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* docs(datasets): add `ibis.FileDataset` to contents

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets)!: expose `load` and `save` publicly

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): remove second filepath assignment

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@ankatiyar @tdhooghe
ci(datasets): Fix mypy errors (kedro-org#893)
27e58cd 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
more strict type hinting and casting for linting
178f0f4 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add back python 3.12 to see if it is compatible
0133025 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
update testing documentation
fc2b3b8 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe tdhooghe force-pushed the feature/save-pd-to-snowflaketable branch 2 times, most recently from a40aa99 to fc2b3b8 Compare October 21, 2024 16:23
@tdhooghe tdhooghe marked this pull request as ready for review October 21, 2024 16:24
merelcht and others added 4 commits October 21, 2024 17:29
@merelcht
Merge branch 'main' into feature/save-pd-to-snowflaketable
c18fc13 
Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
@tdhooghe
remove uv lock
7efd162 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
Merge remote-tracking branch 'refs/remotes/origin/feature/save-pd-to-…
f9f3d40 
…snowflaketable' into feature/save-pd-to-snowflaketable
@tdhooghe
remove unnecessary change to ibis file
620b27a 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
Update kedro-datasets/kedro_datasets/snowflake/snowpark_dataset.py
4d514d9 
Signed-off-by: Thomas <thomas.dhooghe95@gmail.com>
tdhooghe and others added 3 commits October 21, 2024 18:37
@tdhooghe
Update kedro-datasets/kedro_datasets/snowflake/snowpark_dataset.py
2fddf19 
Signed-off-by: Thomas <thomas.dhooghe95@gmail.com>
@tdhooghe
touch as little reqs ass possible
e25be21 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
update changelog
e3bf820 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe tdhooghe changed the title feat(datasets): save pandas df directly to SnowflakeTable feat(datasets): Improved compatibility, functionality and testing for SnowflakeTableDataset Oct 21, 2024
@tdhooghe
bump cloudpickle version to enable snowpark 1.23
1e0f0c4 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tdhooghe @deepyaman @ankatiyar @merelcht