Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataCatalog2.0]: Move pattern resolution logic to the separate component #4123

Merged
merged 89 commits into from
Sep 12, 2024

Commits on Jul 31, 2024

  1. Added a skeleton for AbstractDataCatalog and KedroDataCatalog

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    a8f4fb3 View commit details
    Browse the repository at this point in the history
  2. Removed from_config method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    7d56818 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Configuration menu
    Copy the full SHA
    787e121 View commit details
    Browse the repository at this point in the history
  2. Implemented _init_datasets method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    0b80f23 View commit details
    Browse the repository at this point in the history
  3. Implemented get dataset

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    5c727df View commit details
    Browse the repository at this point in the history
  4. Started resolve_patterns implementation

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    05c9171 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Implemented resolve_patterns

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    5c804d6 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e9ba5c4 View commit details
    Browse the repository at this point in the history
  3. Fixed credentials resolving

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    530f7d6 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Updated match pattern

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    64be83c View commit details
    Browse the repository at this point in the history
  2. Implemented add from dict method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    c29828a View commit details
    Browse the repository at this point in the history
  3. Updated io __init__

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    957403a View commit details
    Browse the repository at this point in the history
  4. Added list method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    14908ff View commit details
    Browse the repository at this point in the history
  5. Implemented _validate_missing_keys

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    c5e925b View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Added datasets access logic

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    b9a92b0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2cb794f View commit details
    Browse the repository at this point in the history
  3. Added __contains__ and comments on lazy loading

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    2f32593 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. Renamed dataset_name to ds_name

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    d1ea64e View commit details
    Browse the repository at this point in the history
  2. Updated some docstrings

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    fb89fca View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Configuration menu
    Copy the full SHA
    4486939 View commit details
    Browse the repository at this point in the history
  2. Fixed _update_ds_configs

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    c667645 View commit details
    Browse the repository at this point in the history
  3. Fixed _init_datasets

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    be8e929 View commit details
    Browse the repository at this point in the history
  4. Implemented add_runtime_patterns

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    ec7ac39 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Fixed runtime patterns usage

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    8e23450 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. Configuration menu
    Copy the full SHA
    529e61a View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Configuration menu
    Copy the full SHA
    e4cb21c View commit details
    Browse the repository at this point in the history
  2. Moved pattern logic out of data catalog, implemented KedroDataCatalog

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    50bc816 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Configuration menu
    Copy the full SHA
    6dfbcb0 View commit details
    Browse the repository at this point in the history
  2. KedroDataCatalog updates

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    9346f08 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. Added property to return config

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    9568e29 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    86efdfe View commit details
    Browse the repository at this point in the history
  3. Added list patterns method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    5e27660 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Renamed and moved ConfigResolver

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    72b11d0 View commit details
    Browse the repository at this point in the history
  2. Renamed ConfigResolver

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    f0a4090 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a4da52a View commit details
    Browse the repository at this point in the history
  4. Cleaned KedroDataCatalog

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    7d6227f View commit details
    Browse the repository at this point in the history
  5. Cleaned up DataCatalogConfigResolver

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    4092291 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Docs build fix attempt

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    63e47f9 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Removed KedroDataCatalog

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    68f6527 View commit details
    Browse the repository at this point in the history
  2. Updated from_config method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    2ac4a2f View commit details
    Browse the repository at this point in the history
  3. Updated constructor and add methods

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    cb5879d View commit details
    Browse the repository at this point in the history
  4. Updated _get_dataset method

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    9038e96 View commit details
    Browse the repository at this point in the history
  5. Updated __contains__

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    cc89565 View commit details
    Browse the repository at this point in the history
  6. Updated __eq__ and shallow_copy

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    59b6764 View commit details
    Browse the repository at this point in the history
  7. Added __iter__ and __getitem__

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    4f5a3fb View commit details
    Browse the repository at this point in the history
  8. Removed unused imports

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    12ed6f2 View commit details
    Browse the repository at this point in the history
  9. Added TODO

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    a106cec View commit details
    Browse the repository at this point in the history
  10. Updated runner.run()

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    6df04f7 View commit details
    Browse the repository at this point in the history
  11. Updated session

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    8566e27 View commit details
    Browse the repository at this point in the history
  12. Added confil_resolver property

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    2dcea33 View commit details
    Browse the repository at this point in the history
  13. Updated catalog list command

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    a46597f View commit details
    Browse the repository at this point in the history
  14. Updated catalog create command

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    3787545 View commit details
    Browse the repository at this point in the history
  15. Updated catalog rank command

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    68d612d View commit details
    Browse the repository at this point in the history
  16. Updated catalog resolve command

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    af5bee9 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    acc4d6e View commit details
    Browse the repository at this point in the history
  18. Remove some methods

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    e67ff0f View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Removed ds configs from catalog

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    7b3afa2 View commit details
    Browse the repository at this point in the history
  2. Fixed lint

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    658a759 View commit details
    Browse the repository at this point in the history
  3. Fixed typo

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    7be2a8e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    09f3f26 View commit details
    Browse the repository at this point in the history
  5. Added module docstring

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    9e43a9a View commit details
    Browse the repository at this point in the history
  6. Removed None from Pattern type

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    25b6501 View commit details
    Browse the repository at this point in the history
  7. Fixed docs failing to find class reference

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    3a646de View commit details
    Browse the repository at this point in the history
  8. Fixed docs failing to find class reference

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    5e5df4a View commit details
    Browse the repository at this point in the history
  9. Updated Patterns type

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    aa59a35 View commit details
    Browse the repository at this point in the history
  10. Fix tests (#4149)

    * Fix most tests
    
    Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
    
    * Fix most tests
    
    Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
    
    ---------
    
    Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
    ankatiyar authored Sep 6, 2024
    Configuration menu
    Copy the full SHA
    c7efa3e View commit details
    Browse the repository at this point in the history
  11. Returned constants to avoid breaking changes

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    023ffc6 View commit details
    Browse the repository at this point in the history
  12. Minor fix

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    585b44f View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Updated test_sorting_order_with_other_dataset_through_extra_pattern

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    e447078 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    beb0165 View commit details
    Browse the repository at this point in the history
  3. Removed odd properties

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    975e968 View commit details
    Browse the repository at this point in the history
  4. Updated tests

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    11d782c View commit details
    Browse the repository at this point in the history
  5. Removed None from _fetch_credentials input

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    e4abd23 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Renamed DataCatalogConfigResolver to CatalogConfigResolver

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    6433dd8 View commit details
    Browse the repository at this point in the history
  2. Renamed _init_configs to _resolve_config_credentials

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    355576f View commit details
    Browse the repository at this point in the history
  3. Moved functions to the class

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    39d9ff6 View commit details
    Browse the repository at this point in the history
  4. Refactored resolve_dataset_pattern

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    659c9da View commit details
    Browse the repository at this point in the history
  5. Fixed refactored part

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    840b32a View commit details
    Browse the repository at this point in the history
  6. Changed the order of arguments for DataCatalog constructor

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    77f551c View commit details
    Browse the repository at this point in the history
  7. Replaced __getitem__ with .get()

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    6e079a1 View commit details
    Browse the repository at this point in the history
  8. Updated catalog commands

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    1f7e5f8 View commit details
    Browse the repository at this point in the history
  9. Moved warm up block outside of the try block

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    80f0e3d View commit details
    Browse the repository at this point in the history
  10. Fixed linter

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    017cda3 View commit details
    Browse the repository at this point in the history
  11. Removed odd copying

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    cab6f06 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Updated release notes

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    8f604d1 View commit details
    Browse the repository at this point in the history
  2. Returned DatasetError

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    9a4db18 View commit details
    Browse the repository at this point in the history
  3. Added _dataset_patterns and _default_pattern to _config_resolver to a…

    …void breaking change
    
    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    0a6946a View commit details
    Browse the repository at this point in the history
  4. Made resolve_dataset_pattern return just dict

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    fee7bd6 View commit details
    Browse the repository at this point in the history
  5. Fixed linter

    Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
    ElenaKhaustova committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f5a7992 View commit details
    Browse the repository at this point in the history