Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(datasets): Refactored the ManagedTableDataset by Separating Out Common Table Logic #827

Merged
merged 55 commits into from
Oct 10, 2024
Merged

feat(datasets): Refactored the ManagedTableDataset by Separating Out Common Table Logic #827

merged 55 commits into from
Oct 10, 2024

Commits on Sep 8, 2024

  1. added a base table implementation 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    fd0de72 View commit details
    Browse the repository at this point in the history

  2. added a base table dataset implementation 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    0392aba View commit details
    Browse the repository at this point in the history

  3. renamed the module with the base classes 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    38c470f View commit details
    Browse the repository at this point in the history

  4. refactored ManagedTable using BaseTable 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    d424829 View commit details
    Browse the repository at this point in the history

  5. refactored ManagedTableDataset using BaseTableDataset 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    7091986 View commit details
    Browse the repository at this point in the history

  6. removed primary_key attr from BaseTable 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    081d5d1 View commit details
    Browse the repository at this point in the history

  7. updated the format attrs of ManagedTable 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    e2cd580 View commit details
    Browse the repository at this point in the history

  8. implemented the _load() method of ManagedTableDataset 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    33814be View commit details
    Browse the repository at this point in the history

  9. removed unnecessary imports 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    6f91dc8 View commit details
    Browse the repository at this point in the history

  10. reorganized the attrs of BaseTable 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    410cf4a View commit details
    Browse the repository at this point in the history

  11. implemented create_table() in ManagedTableDataset 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    1bf0b8b View commit details
    Browse the repository at this point in the history

  12. added the version attr to BaseTableDataset and updated load() 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    ee7e073 View commit details
    Browse the repository at this point in the history

  13. updated the base and managed datasets with all attrs 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    d87f4ed View commit details
    Browse the repository at this point in the history

  14. updated the supported formats 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    a4feeff View commit details
    Browse the repository at this point in the history

  15. added external table and external table dataset implementations 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    0db5639 View commit details
    Browse the repository at this point in the history

  16. added a val func to check for format when using upsert mode 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    e19eba4 View commit details
    Browse the repository at this point in the history

  17. imported the ExternalTableDataset into the main pkg 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    2c46978 View commit details
    Browse the repository at this point in the history

  18. improved the docstrings in the code 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    2777ea1 View commit details
    Browse the repository at this point in the history

  19. added format to the _describe() 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    6e1b17e View commit details
    Browse the repository at this point in the history

  20. updated the save methods to incorporate partition columns 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    039f9e0 View commit details
    Browse the repository at this point in the history

  21. reverted the default write_mode back to None 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    fb87133 View commit details
    Browse the repository at this point in the history

  22. extended the _validate_write_mode() func to include formats 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    b7a5c33 View commit details
    Browse the repository at this point in the history

  23. updated the save() logic to work with single or multiple partition cols 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    577dd91 View commit details
    Browse the repository at this point in the history

  24. updated the docstrings for the datasets with missing attrs 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    6e729e3 View commit details
    Browse the repository at this point in the history

  25. introduced a location attr for creating ext tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    cbb8514 View commit details
    Browse the repository at this point in the history

  26. updated the save funcs to incorporate the locations attr 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    4a2dc6e View commit details
    Browse the repository at this point in the history

  27. moved the func to check if table exists to BaseTable 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    1734f44 View commit details
    Browse the repository at this point in the history

  28. added a val func to check if location is provided if table does not e… 

    …xist

Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    b03bee8 View commit details
    Browse the repository at this point in the history

  29. moved the val func for checking if write_mode supported to ExternalTable 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    dc3550e View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. removed the func for adding options to writer for better readability 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    78eee4d View commit details
    Browse the repository at this point in the history

  2. added a validation check for overwrites on ext tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    09a8cb2 View commit details
    Browse the repository at this point in the history

  3. implemented the _save_overwrite() func for ext tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    7e493c2 View commit details
    Browse the repository at this point in the history

  4. removed mentions of a default write mode 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    6343226 View commit details
    Browse the repository at this point in the history

  5. improved the docstrings 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    ccc60c4 View commit details
    Browse the repository at this point in the history

  6. fixed lint issues 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    99202ee View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. fixed a couple of bugs in the Table classes 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    31f385a View commit details
    Browse the repository at this point in the history

  2. updated the _save_overwrite() logic for ext tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    5016dee View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. renamed the val funcs of the ext tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    77afc50 View commit details
    Browse the repository at this point in the history

  2. updated _save_overwrite() of ext tables to handle no existing tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    e7ba0e3 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. fixed bug in supporting string partition cols 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    d6ba426 View commit details
    Browse the repository at this point in the history

  2. removed the external table dataset 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    73dc1eb View commit details
    Browse the repository at this point in the history

  3. removed irrelevant attrs from describe() for managed tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    fef688f View commit details
    Browse the repository at this point in the history

  4. preserved order of args 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8ee9ea8 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. updated the tests for base table dataset and managed table dataset 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    4cd7c0f View commit details
    Browse the repository at this point in the history

  2. initialized the base table in the base table dataset 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9982817 View commit details
    Browse the repository at this point in the history

  3. fixed lint issues 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    5a05a34 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feature/databricks_external_dataset

    @MinuraPunchihewa
    MinuraPunchihewa authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    35bbe3b View commit details
    Browse the repository at this point in the history

  5. fixed an incorrect type hint 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    c5d68ea View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Merge branch 'main' into feature/databricks_external_dataset 

    Signed-off-by: Minura Punchihewa <49385643+MinuraPunchihewa@users.noreply.github.com>
    @MinuraPunchihewa
    MinuraPunchihewa authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    e651008 View commit details
    Browse the repository at this point in the history

  2. removed redundant check on save() 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    b6dc5b9 View commit details
    Browse the repository at this point in the history

  3. added the missing unit tests 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    c5ea9d9 View commit details
    Browse the repository at this point in the history

  4. fixed the tests for saving external tables 

    Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    0558d96 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Merge branch 'main' into feature/databricks_external_dataset 

    Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com>
    @ankatiyar
    ankatiyar authored Oct 10, 2024
    Configuration menu
    Copy the full SHA
    145a2cf View commit details
    Browse the repository at this point in the history

  2. lint 

    Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
    @ankatiyar
    ankatiyar committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    d598d79 View commit details
    Browse the repository at this point in the history

  3. Merge main 

    Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
    @ankatiyar
    ankatiyar committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    e34715a View commit details
    Browse the repository at this point in the history