diff --git a/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/how_to_instantiate_a_specific_filesystem_data_context.md b/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/how_to_instantiate_a_specific_filesystem_data_context.md index 7a7c3dbeac3a..3c553455e70e 100644 --- a/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/how_to_instantiate_a_specific_filesystem_data_context.md +++ b/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/how_to_instantiate_a_specific_filesystem_data_context.md @@ -36,18 +36,35 @@ If you are using GX for multiple projects you may wish to utilize a different Da Each Filesystem Data Context has a root folder in which it was initialized. This root folder will be used to indicate which specific Filesystem Data Context should be instantiated. -```python name="tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py path_to_context_root_folder" +```python name="tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py path_to_project_root" ``` ### 2. Run GX's `get_context(...)` method -We provide our Filesystem Data Context's root folder path to the GX library's `get_context(...)` method as the `context_root_dir` parameter. Because we are providing a path to an existing Data Context, the `get_context(...)` method will instantiate and return the Data Context at that location. +We provide our Filesystem Data Context's root folder path to the GX library's `get_context(...)` method as the `project_root_dir` parameter. Because we are providing a path to an existing Data Context, the `get_context(...)` method will instantiate and return the Data Context at that location. ```python name="tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py get_filesystem_data_context" ``` +:::info Project root vs context root +Note that there is a subtle distinction between the `project_root_dir` and `context_root_dir` arguments accepted by `get_context(...)`. + +Your context root is the directory that contains all your GX config while your project root refers to your actual working directory (and therefore contains the context root). + +```bash +# The overall directory is your project root +data/ +great_expectations/ # The GX folder with your config is your context root + great_expectations.yml + ... +... +``` + +Both are functionally equivalent for purposes of working with a file-backed project. +::: + :::info What if the folder does not contain a Data Context? -If the `context_root_dir` provided to the `get_context(...)` method points to a folder that does not already have a Data Context present, the `get_context(...)` method will initialize a new Filesystem Data Context at that location. +If the root directory provided to the `get_context(...)` method points to a folder that does not already have a Data Context present, the `get_context(...)` method will initialize a new Filesystem Data Context at that location. The `get_context(...)` method will then instantiate and return the newly initialized Data Context. ::: diff --git a/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/instantiate_data_context.md b/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/instantiate_data_context.md index 03a46f5dcbfb..bcd80fd1224f 100644 --- a/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/instantiate_data_context.md +++ b/docs/docusaurus/docs/guides/setup/configuring_data_contexts/instantiating_data_contexts/instantiate_data_context.md @@ -132,18 +132,35 @@ If you're using GX for multiple projects, you might want to use a different Data Each Filesystem Data Context has a root folder in which it was initialized. This root folder identifies the specific Filesystem Data Context to instantiate. -```python name="tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py path_to_context_root_folder" +```python name="tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py path_to_project_root" ``` ### Run the `get_context(...)` method -You provide the path for your empty folder to the GX library's `get_context(...)` method as the `context_root_dir` parameter. Because you are providing a path to an empty folder, the `get_context(...)` method instantiates and return the Data Context at that location. +You provide the path for your empty folder to the GX library's `get_context(...)` method as the `project_root_dir` parameter. Because you are providing a path to an empty folder, the `get_context(...)` method instantiates and return the Data Context at that location. ```python name="tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py get_filesystem_data_context" ``` +:::info Project root vs context root +Note that there is a subtle distinction between the `project_root_dir` and `context_root_dir` arguments accepted by `get_context(...)`. + +Your context root is the directory that contains all your GX config while your project root refers to your actual working directory (and therefore contains the context root). + +```bash +# The overall directory is your project root +data/ +great_expectations/ # The GX folder with your config is your context root + great_expectations.yml + ... +... +``` + +Both are functionally equivalent for purposes of working with a file-backed project. +::: + :::info What if the folder does not contain a Data Context? -If the `context_root_dir` provided to the `get_context(...)` method points to a folder that does not already have a Data Context, the `get_context(...)` method initializes a new Filesystem Data Context in that location. +If the root directory provided to the `get_context(...)` method points to a folder that does not already have a Data Context, the `get_context(...)` method initializes a new Filesystem Data Context in that location. The `get_context(...)` method instantiates and returns the newly initialized Data Context. ::: diff --git a/great_expectations/data_context/data_context/abstract_data_context.py b/great_expectations/data_context/data_context/abstract_data_context.py index d0683b7ede92..51d825f21194 100644 --- a/great_expectations/data_context/data_context/abstract_data_context.py +++ b/great_expectations/data_context/data_context/abstract_data_context.py @@ -1172,8 +1172,8 @@ def get_config_with_variables_substituted( ) -> DataContextConfig: """ Substitute vars in config of form ${var} or $(var) with values found in the following places, - in order of precedence: ge_cloud_config (for Data Contexts in GX Cloud mode), runtime_environment, - environment variables, config_variables, or ge_cloud_config_variable_defaults (allows certain variables to + in order of precedence: gx_cloud_config (for Data Contexts in GX Cloud mode), runtime_environment, + environment variables, config_variables, or gx_cloud_config_variable_defaults (allows certain variables to be optional in GX Cloud mode). """ if not config: diff --git a/great_expectations/data_context/data_context/cloud_data_context.py b/great_expectations/data_context/data_context/cloud_data_context.py index ae86881caa37..90be9c401b0f 100644 --- a/great_expectations/data_context/data_context/cloud_data_context.py +++ b/great_expectations/data_context/data_context/cloud_data_context.py @@ -11,7 +11,6 @@ Mapping, Optional, Sequence, - Tuple, Union, cast, overload, @@ -99,14 +98,11 @@ def __init__( # noqa: PLR0913 self, project_config: Optional[Union[DataContextConfig, Mapping]] = None, context_root_dir: Optional[PathStr] = None, + project_root_dir: Optional[PathStr] = None, runtime_environment: Optional[dict] = None, cloud_base_url: Optional[str] = None, cloud_access_token: Optional[str] = None, cloud_organization_id: Optional[str] = None, - # Deprecated as of 0.15.37 - ge_cloud_base_url: Optional[str] = None, - ge_cloud_access_token: Optional[str] = None, - ge_cloud_organization_id: Optional[str] = None, ) -> None: """ CloudDataContext constructor @@ -117,20 +113,6 @@ def __init__( # noqa: PLR0913 config_variables.yml and the environment cloud_config (GXCloudConfig): GXCloudConfig corresponding to current CloudDataContext """ - # Chetan - 20221208 - not formally deprecating these values until a future date - ( - cloud_base_url, - cloud_access_token, - cloud_organization_id, - ) = CloudDataContext._resolve_cloud_args( - cloud_base_url=cloud_base_url, - cloud_access_token=cloud_access_token, - cloud_organization_id=cloud_organization_id, - ge_cloud_base_url=ge_cloud_base_url, - ge_cloud_access_token=ge_cloud_access_token, - ge_cloud_organization_id=ge_cloud_organization_id, - ) - self._check_if_latest_version() self._cloud_config = self.get_cloud_config( cloud_base_url=cloud_base_url, @@ -138,7 +120,8 @@ def __init__( # noqa: PLR0913 cloud_organization_id=cloud_organization_id, ) self._context_root_directory = self.determine_context_root_directory( - context_root_dir + context_root_dir=context_root_dir, + project_root_dir=project_root_dir, ) self._project_config = self._init_project_config(project_config) @@ -173,31 +156,6 @@ def _initialize_usage_statistics( # Usage statistics are always disabled within Cloud-backed environments. self._usage_statistics_handler = None - @staticmethod - def _resolve_cloud_args( # noqa: PLR0913 - cloud_base_url: Optional[str] = None, - cloud_access_token: Optional[str] = None, - cloud_organization_id: Optional[str] = None, - # Deprecated as of 0.15.37 - ge_cloud_base_url: Optional[str] = None, - ge_cloud_access_token: Optional[str] = None, - ge_cloud_organization_id: Optional[str] = None, - ) -> Tuple[Optional[str], Optional[str], Optional[str]]: - cloud_base_url = ( - cloud_base_url if cloud_base_url is not None else ge_cloud_base_url - ) - cloud_access_token = ( - cloud_access_token - if cloud_access_token is not None - else ge_cloud_access_token - ) - cloud_organization_id = ( - cloud_organization_id - if cloud_organization_id is not None - else ge_cloud_organization_id - ) - return cloud_base_url, cloud_access_token, cloud_organization_id - @override def _register_providers(self, config_provider: _ConfigurationProvider) -> None: """ @@ -247,8 +205,13 @@ def is_cloud_config_available( @classmethod def determine_context_root_directory( - cls, context_root_dir: Optional[PathStr] + cls, + context_root_dir: Optional[PathStr], + project_root_dir: Optional[PathStr], ) -> str: + context_root_dir = cls._resolve_context_root_dir_and_project_root_dir( + context_root_dir=context_root_dir, project_root_dir=project_root_dir + ) if context_root_dir is None: context_root_dir = os.getcwd() # noqa: PTH109 logger.info( diff --git a/great_expectations/data_context/data_context/data_context.py b/great_expectations/data_context/data_context/data_context.py index 0d3f5f0acc06..38f6983372f5 100644 --- a/great_expectations/data_context/data_context/data_context.py +++ b/great_expectations/data_context/data_context/data_context.py @@ -34,7 +34,7 @@ def DataContext( # noqa: PLR0913 cloud_access_token: None = ..., cloud_organization_id: None = ..., ) -> FileDataContext: - # If `context_root_dir` is provided and `cloud_mode`/`ge_cloud_mode` are `False` a `FileDataContext` will always be returned. + # If `context_root_dir` is provided and `cloud_mode` is `False` a `FileDataContext` will always be returned. ... @@ -169,7 +169,8 @@ def _init_context_root_directory( ) -> str: if cloud_mode and context_root_dir is None: context_root_dir = CloudDataContext.determine_context_root_directory( - context_root_dir + context_root_dir=context_root_dir, + project_root_dir=None, ) else: context_root_dir = ( diff --git a/great_expectations/data_context/data_context/file_data_context.py b/great_expectations/data_context/data_context/file_data_context.py index c2076143a0d7..ccde67645238 100644 --- a/great_expectations/data_context/data_context/file_data_context.py +++ b/great_expectations/data_context/data_context/file_data_context.py @@ -45,6 +45,7 @@ def __init__( self, project_config: Optional[DataContextConfig] = None, context_root_dir: Optional[PathStr] = None, + project_root_dir: Optional[PathStr] = None, runtime_environment: Optional[dict] = None, ) -> None: """FileDataContext constructor @@ -57,7 +58,8 @@ def __init__( config_variables.yml and the environment """ self._context_root_directory = self._init_context_root_directory( - context_root_dir + context_root_dir=context_root_dir, + project_root_dir=project_root_dir, ) self._scaffold_project() @@ -67,16 +69,18 @@ def __init__( runtime_environment=runtime_environment, ) - def _init_context_root_directory(self, context_root_dir: Optional[PathStr]) -> str: + def _init_context_root_directory( + self, context_root_dir: Optional[PathStr], project_root_dir: Optional[PathStr] + ) -> str: + context_root_dir = self._resolve_context_root_dir_and_project_root_dir( + context_root_dir=context_root_dir, project_root_dir=project_root_dir + ) + if isinstance(context_root_dir, pathlib.Path): context_root_dir = str(context_root_dir) if not context_root_dir: - context_root_dir = FileDataContext.find_context_root_dir() - if not context_root_dir: - raise ValueError( - "A FileDataContext relies on the presence of a local great_expectations.yml project config" - ) + context_root_dir = self.find_context_root_dir() return context_root_dir diff --git a/great_expectations/data_context/data_context/serializable_data_context.py b/great_expectations/data_context/data_context/serializable_data_context.py index 0fa3362780ad..7f62c78de791 100644 --- a/great_expectations/data_context/data_context/serializable_data_context.py +++ b/great_expectations/data_context/data_context/serializable_data_context.py @@ -93,6 +93,23 @@ def _save_project_config(self, _fds_datasource=None) -> None: """ raise NotImplementedError + @classmethod + def _resolve_context_root_dir_and_project_root_dir( + cls, context_root_dir: PathStr | None, project_root_dir: PathStr | None + ) -> PathStr | None: + if project_root_dir and context_root_dir: + raise TypeError( + "'project_root_dir' and 'context_root_dir' are conflicting args; please only provide one" + ) + + if project_root_dir: + project_root_dir = pathlib.Path(project_root_dir).absolute() + context_root_dir = pathlib.Path(project_root_dir) / cls.GX_DIR + elif context_root_dir: + context_root_dir = pathlib.Path(context_root_dir).absolute() + + return context_root_dir + def _check_for_usage_stats_sync( # noqa: PLR0911 self, project_config: DataContextConfig ) -> bool: diff --git a/great_expectations/util.py b/great_expectations/util.py index fa8e4bea26d2..70bc0bd7ac41 100644 --- a/great_expectations/util.py +++ b/great_expectations/util.py @@ -10,7 +10,6 @@ import json import logging import os -import pathlib import pstats import re import sys @@ -60,7 +59,7 @@ from great_expectations.compatibility import sqlalchemy from great_expectations.compatibility.sqlalchemy import sqlalchemy as sa from great_expectations.compatibility.typing_extensions import override -from great_expectations.core._docs_decorators import deprecated_argument, public_api +from great_expectations.core._docs_decorators import public_api from great_expectations.exceptions import ( GXCloudConfigurationError, PluginClassNotFoundError, @@ -1741,19 +1740,29 @@ def convert_pandas_series_decimal_to_float_dtype( @overload -def get_context( # type: ignore[misc] # overlapping overload false positive? # noqa: PLR0913 +def get_context( # type: ignore[misc] # noqa: PLR0913 project_config: DataContextConfig | Mapping | None = ..., - context_root_dir: PathStr = ..., + context_root_dir: PathStr = ..., # If context_root_dir is provided, project_root_dir shouldn't be + project_root_dir: None = ..., + runtime_environment: dict | None = ..., + cloud_base_url: None = ..., + cloud_access_token: None = ..., + cloud_organization_id: None = ..., + cloud_mode: Literal[False] | None = ..., +) -> FileDataContext: + ... + + +@overload +def get_context( # type: ignore[misc] # noqa: PLR0913 + project_config: DataContextConfig | Mapping | None = ..., + context_root_dir: None = ..., + project_root_dir: PathStr = ..., # If project_root_dir is provided, context_root_dir shouldn't be runtime_environment: dict | None = ..., cloud_base_url: None = ..., cloud_access_token: None = ..., cloud_organization_id: None = ..., cloud_mode: Literal[False] | None = ..., - # Deprecated as of 0.15.37 - ge_cloud_base_url: None = ..., - ge_cloud_access_token: None = ..., - ge_cloud_organization_id: None = ..., - ge_cloud_mode: Literal[False] | None = ..., ) -> FileDataContext: ... @@ -1762,16 +1771,12 @@ def get_context( # type: ignore[misc] # overlapping overload false positive? # def get_context( # noqa: PLR0913 project_config: DataContextConfig | Mapping | None = ..., context_root_dir: None = ..., + project_root_dir: None = ..., runtime_environment: dict | None = ..., cloud_base_url: str | None = ..., cloud_access_token: str | None = ..., cloud_organization_id: str | None = ..., cloud_mode: Literal[True] = ..., - # Deprecated as of 0.15.37 - ge_cloud_base_url: str | None = ..., - ge_cloud_access_token: str | None = ..., - ge_cloud_organization_id: str | None = ..., - ge_cloud_mode: bool | None = ..., ) -> CloudDataContext: ... @@ -1780,38 +1785,26 @@ def get_context( # noqa: PLR0913 def get_context( # noqa: PLR0913 project_config: DataContextConfig | Mapping | None = ..., context_root_dir: PathStr | None = ..., + project_root_dir: PathStr | None = ..., runtime_environment: dict | None = ..., cloud_base_url: str | None = ..., cloud_access_token: str | None = ..., cloud_organization_id: str | None = ..., cloud_mode: bool | None = ..., - # Deprecated as of 0.15.37 - ge_cloud_base_url: str | None = ..., - ge_cloud_access_token: str | None = ..., - ge_cloud_organization_id: str | None = ..., - ge_cloud_mode: bool | None = ..., ) -> AbstractDataContext: ... @public_api -@deprecated_argument(argument_name="ge_cloud_base_url", version="0.15.37") -@deprecated_argument(argument_name="ge_cloud_access_token", version="0.15.37") -@deprecated_argument(argument_name="ge_cloud_organization_id", version="0.15.37") -@deprecated_argument(argument_name="ge_cloud_mode", version="0.15.37") def get_context( # noqa: PLR0913 project_config: DataContextConfig | Mapping | None = None, context_root_dir: PathStr | None = None, + project_root_dir: PathStr | None = None, runtime_environment: dict | None = None, cloud_base_url: str | None = None, cloud_access_token: str | None = None, cloud_organization_id: str | None = None, cloud_mode: bool | None = None, - # Deprecated as of 0.15.37 - ge_cloud_base_url: str | None = None, - ge_cloud_access_token: str | None = None, - ge_cloud_organization_id: str | None = None, - ge_cloud_mode: bool | None = None, ) -> AbstractDataContext: """Method to return the appropriate Data Context depending on parameters and environment. @@ -1823,9 +1816,7 @@ def get_context( # noqa: PLR0913 This method returns the appropriate Data Context based on which parameters you've passed and / or your environment configuration: - FileDataContext: Configuration stored in a file. - - EphemeralDataContext: Configuration passed in at runtime. - - CloudDataContext: Configuration stored in Great Expectations Cloud. Read on for more details about each of the Data Context types: @@ -1833,17 +1824,14 @@ def get_context( # noqa: PLR0913 **FileDataContext:** A Data Context configured via a yaml file. Returned by default if you have no cloud configuration set up and pass no parameters. If you pass context_root_dir, we will look for a great_expectations.yml configuration there. If not we will look at the following locations: - Path defined in a GX_HOME environment variable. - - The current directory. - - Parent directories of the current directory (e.g. in case you invoke the CLI in a sub folder of your Great Expectations directory). Relevant parameters + - project_root_dir: Provide the project root where a GX project exists or will be scaffolded (contains the context root). - context_root_dir: Provide an alternative directory to look for GX config. - - project_config: Optionally override the configuration on disk - only if `context_root_dir` is also provided. - - runtime_environment: Optionally override specific configuration values. **EphemeralDataContext:** A temporary, in-memory Data Context typically used in a pipeline. The default if you pass in only a project_config and have no cloud configuration set up. @@ -1851,7 +1839,6 @@ def get_context( # noqa: PLR0913 Relevant parameters - project_config: Used to configure the Data Context. - - runtime_environment: Optionally override specific configuration values. **CloudDataContext:** A Data Context whose configuration comes from Great Expectations Cloud. The default if you have a cloud configuration set up. Pass `cloud_mode=False` if you have a cloud configuration set up and you do not wish to create a CloudDataContext. @@ -1861,20 +1848,16 @@ def get_context( # noqa: PLR0913 Relevant parameters - cloud_base_url: Override env var or great_expectations.conf file. - - cloud_access_token: Override env var or great_expectations.conf file. - - cloud_organization_id: Override env var or great_expectations.conf file. - - cloud_mode: Set to True or False to explicitly enable/disable cloud mode. - - project_config: Optionally override the cloud configuration. - - runtime_environment: Optionally override specific configuration values. Args: project_config: In-memory configuration for Data Context. context_root_dir (str or pathlib.Path): Path to directory that contains great_expectations.yml file + project_root_dir (str or pathlib.Path): Path to project root (contains context root with GX config) runtime_environment: A dictionary of values can be passed to a DataContext when it is instantiated. These values will override both values from the config variables file and from environment variables. @@ -1883,11 +1866,6 @@ def get_context( # noqa: PLR0913 cloud_organization_id: org_id for GX Cloud account. cloud_mode: whether to run GX in Cloud mode (default is None). If None, cloud mode is assumed if cloud credentials are set up. Set to False to override. - ge_cloud_base_url: url for GX Cloud endpoint. - ge_cloud_access_token: access_token for GX Cloud account. - ge_cloud_organization_id: org_id for GX Cloud account. - ge_cloud_mode: whether to run GX in Cloud mode (default is None). - If None, cloud mode is assumed if cloud credentials are set up. Set to False to override. Returns: A Data Context. Either a FileDataContext, EphemeralDataContext, or @@ -1903,15 +1881,12 @@ def get_context( # noqa: PLR0913 cloud_context = _get_cloud_context( project_config=project_config, context_root_dir=context_root_dir, + project_root_dir=project_root_dir, runtime_environment=runtime_environment, cloud_mode=cloud_mode, cloud_base_url=cloud_base_url, cloud_access_token=cloud_access_token, cloud_organization_id=cloud_organization_id, - ge_cloud_mode=ge_cloud_mode, - ge_cloud_base_url=ge_cloud_base_url, - ge_cloud_access_token=ge_cloud_access_token, - ge_cloud_organization_id=ge_cloud_organization_id, ) if cloud_context: return cloud_context @@ -1920,6 +1895,7 @@ def get_context( # noqa: PLR0913 file_context = _get_file_context( project_config=project_config, context_root_dir=context_root_dir, + project_root_dir=project_root_dir, runtime_environment=runtime_environment, ) if file_context: @@ -1953,36 +1929,15 @@ def _prepare_project_config( def _get_cloud_context( # noqa: PLR0913 project_config: DataContextConfig | Mapping | None = None, context_root_dir: PathStr | None = None, + project_root_dir: PathStr | None = None, runtime_environment: dict | None = None, cloud_base_url: str | None = None, cloud_access_token: str | None = None, cloud_organization_id: str | None = None, cloud_mode: bool | None = None, - # Deprecated as of 0.15.37 - ge_cloud_base_url: str | None = None, - ge_cloud_access_token: str | None = None, - ge_cloud_organization_id: str | None = None, - ge_cloud_mode: bool | None = None, ) -> CloudDataContext | None: from great_expectations.data_context.data_context import CloudDataContext - # Chetan - 20221208 - not formally deprecating these values until a future date - ( - cloud_base_url, - cloud_access_token, - cloud_organization_id, - cloud_mode, - ) = _resolve_cloud_args( - cloud_mode=cloud_mode, - cloud_base_url=cloud_base_url, - cloud_access_token=cloud_access_token, - cloud_organization_id=cloud_organization_id, - ge_cloud_mode=ge_cloud_mode, - ge_cloud_base_url=ge_cloud_base_url, - ge_cloud_access_token=ge_cloud_access_token, - ge_cloud_organization_id=ge_cloud_organization_id, - ) - config_available = CloudDataContext.is_cloud_config_available( cloud_base_url=cloud_base_url, cloud_access_token=cloud_access_token, @@ -1995,6 +1950,7 @@ def _get_cloud_context( # noqa: PLR0913 project_config=project_config, runtime_environment=runtime_environment, context_root_dir=context_root_dir, + project_root_dir=project_root_dir, cloud_base_url=cloud_base_url, cloud_access_token=cloud_access_token, cloud_organization_id=cloud_organization_id, @@ -2008,52 +1964,24 @@ def _get_cloud_context( # noqa: PLR0913 return None -def _resolve_cloud_args( # noqa: PLR0913 - cloud_base_url: str | None = None, - cloud_access_token: str | None = None, - cloud_organization_id: str | None = None, - cloud_mode: bool | None = None, - # Deprecated as of 0.15.37 - ge_cloud_base_url: str | None = None, - ge_cloud_access_token: str | None = None, - ge_cloud_organization_id: str | None = None, - ge_cloud_mode: bool | None = None, -) -> tuple[str | None, str | None, str | None, bool | None]: - cloud_base_url = cloud_base_url if cloud_base_url is not None else ge_cloud_base_url - cloud_access_token = ( - cloud_access_token if cloud_access_token is not None else ge_cloud_access_token - ) - cloud_organization_id = ( - cloud_organization_id - if cloud_organization_id is not None - else ge_cloud_organization_id - ) - cloud_mode = cloud_mode if cloud_mode is not None else ge_cloud_mode - return cloud_base_url, cloud_access_token, cloud_organization_id, cloud_mode - - def _get_file_context( project_config: DataContextConfig | None = None, context_root_dir: PathStr | None = None, + project_root_dir: PathStr | None = None, runtime_environment: dict | None = None, ) -> FileDataContext | None: from great_expectations.data_context.data_context import FileDataContext - if not context_root_dir: - try: - context_root_dir = FileDataContext.find_context_root_dir() - except gx_exceptions.ConfigNotFoundError: - logger.info("Could not find local context root directory") - - if context_root_dir: - context_root_dir = pathlib.Path(context_root_dir).absolute() + try: return FileDataContext( project_config=project_config, context_root_dir=context_root_dir, + project_root_dir=project_root_dir, runtime_environment=runtime_environment, ) - - return None + except gx_exceptions.ConfigNotFoundError: + logger.info("Could not find local file-backed GX project") + return None def _get_ephemeral_context( diff --git a/tests/conftest.py b/tests/conftest.py index 6a15c9a15c36..bb94b557eb92 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -3655,9 +3655,9 @@ def empty_cloud_data_context( cloud_data_context: CloudDataContext = CloudDataContext( project_config=empty_ge_cloud_data_context_config, context_root_dir=project_path_name, - ge_cloud_base_url=ge_cloud_config.base_url, - ge_cloud_access_token=ge_cloud_config.access_token, - ge_cloud_organization_id=ge_cloud_config.organization_id, + cloud_base_url=ge_cloud_config.base_url, + cloud_access_token=ge_cloud_config.access_token, + cloud_organization_id=ge_cloud_config.organization_id, ) return cloud_data_context diff --git a/tests/data_context/test_data_context_config_errors.py b/tests/data_context/test_data_context_config_errors.py index 1ca4aeff7b22..dd3252deb646 100644 --- a/tests/data_context/test_data_context_config_errors.py +++ b/tests/data_context/test_data_context_config_errors.py @@ -7,8 +7,8 @@ import pytest import great_expectations.exceptions as gx_exceptions +from great_expectations.data_context.data_context import DataContext from great_expectations.data_context.util import file_relative_path -from great_expectations.util import get_context pytestmark = pytest.mark.filesystem @@ -32,19 +32,19 @@ def _build_local_dir(fixture_subdir: str | None = None) -> pathlib.Path: def test_DataContext_raises_error_on_config_not_found(build_local_dir: Callable): local_dir = build_local_dir() with pytest.raises(gx_exceptions.ConfigNotFoundError): - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) def test_DataContext_raises_error_on_unparsable_yaml_file(build_local_dir: Callable): local_dir = build_local_dir("bad_yml") with pytest.raises(gx_exceptions.InvalidConfigurationYamlError): - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) def test_DataContext_raises_error_on_invalid_top_level_type(build_local_dir: Callable): local_dir = build_local_dir("invalid_top_level_value_type") with pytest.raises(gx_exceptions.InvalidDataContextConfigError) as exc: - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) error_messages = [] if exc.value.messages and len(exc.value.messages) > 0: @@ -58,7 +58,7 @@ def test_DataContext_raises_error_on_invalid_top_level_type(build_local_dir: Cal def test_DataContext_raises_error_on_invalid_config_version(build_local_dir: Callable): local_dir = build_local_dir("invalid_config_version") with pytest.raises(gx_exceptions.InvalidDataContextConfigError) as exc: - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) error_messages = [] if exc.value.messages and len(exc.value.messages) > 0: @@ -72,7 +72,7 @@ def test_DataContext_raises_error_on_invalid_config_version(build_local_dir: Cal def test_DataContext_raises_error_on_old_config_version(build_local_dir: Callable): local_dir = build_local_dir("old_config_version") with pytest.raises(gx_exceptions.InvalidDataContextConfigError) as exc: - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) assert "Error while processing DataContextConfig" in exc.value.message @@ -82,7 +82,7 @@ def test_DataContext_raises_error_on_missing_config_version_aka_version_zero( ): local_dir = build_local_dir("version_zero") with pytest.raises(gx_exceptions.InvalidDataContextConfigError): - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) def test_DataContext_raises_error_on_missing_config_version_aka_version_zero_with_v2_config( @@ -90,4 +90,4 @@ def test_DataContext_raises_error_on_missing_config_version_aka_version_zero_wit ): local_dir = build_local_dir("version_2-0_but_no_version_defined") with pytest.raises(gx_exceptions.InvalidDataContextConfigError): - get_context(context_root_dir=local_dir) + DataContext(context_root_dir=local_dir) diff --git a/tests/data_context/test_data_context_deprecation.py b/tests/data_context/test_data_context_deprecation.py index 0cd443c18309..a66ae60e792b 100644 --- a/tests/data_context/test_data_context_deprecation.py +++ b/tests/data_context/test_data_context_deprecation.py @@ -6,13 +6,9 @@ from great_expectations.data_context.data_context import ( AbstractDataContext, - CloudDataContext, -) -from great_expectations.data_context.data_context.ephemeral_data_context import ( EphemeralDataContext, ) from great_expectations.data_context.types.base import GXCloudConfig -from great_expectations.util import _resolve_cloud_args as get_context_resolver # Globally scoped so we can reuse across test parameterization cloud_base_url = "my_cloud_url" @@ -33,129 +29,6 @@ ) -@pytest.mark.cloud -@pytest.mark.parametrize( - "cloud_args,expected_resolved_args", - [ - pytest.param( - { - "cloud_base_url": cloud_base_url, - "cloud_access_token": cloud_access_token, - "cloud_organization_id": cloud_organization_id, - }, - (cloud_base_url, cloud_access_token, cloud_organization_id), - id="new_style_args", - ), - pytest.param( - { - "ge_cloud_base_url": ge_cloud_base_url, - "ge_cloud_access_token": ge_cloud_access_token, - "ge_cloud_organization_id": ge_cloud_organization_id, - }, - (ge_cloud_base_url, ge_cloud_access_token, ge_cloud_organization_id), - id="deprecated_args", - ), - pytest.param( - { - "cloud_base_url": cloud_base_url, - "cloud_access_token": cloud_access_token, - "cloud_organization_id": cloud_organization_id, - "ge_cloud_base_url": ge_cloud_base_url, - "ge_cloud_access_token": ge_cloud_access_token, - "ge_cloud_organization_id": ge_cloud_organization_id, - }, - (cloud_base_url, cloud_access_token, cloud_organization_id), - id="conflicting_args", - ), - ], -) -def test_CloudDataContext_resolve_cloud_args( - cloud_args: dict, expected_resolved_args: tuple -): - actual_resolved_args = CloudDataContext._resolve_cloud_args(**cloud_args) - assert actual_resolved_args == expected_resolved_args - - -@pytest.mark.cloud -@pytest.mark.parametrize( - "cloud_args,expected_resolved_args", - [ - pytest.param( - { - "cloud_mode": True, - "cloud_base_url": cloud_base_url, - "cloud_access_token": cloud_access_token, - "cloud_organization_id": cloud_organization_id, - }, - (cloud_base_url, cloud_access_token, cloud_organization_id, True), - id="new_style_args", - ), - pytest.param( - { - "ge_cloud_mode": True, - "ge_cloud_base_url": ge_cloud_base_url, - "ge_cloud_access_token": ge_cloud_access_token, - "ge_cloud_organization_id": ge_cloud_organization_id, - }, - (ge_cloud_base_url, ge_cloud_access_token, ge_cloud_organization_id, True), - id="deprecated_args", - ), - pytest.param( - { - "cloud_mode": True, - "cloud_base_url": cloud_base_url, - "cloud_access_token": cloud_access_token, - "cloud_organization_id": cloud_organization_id, - "ge_cloud_mode": True, - "ge_cloud_base_url": ge_cloud_base_url, - "ge_cloud_access_token": ge_cloud_access_token, - "ge_cloud_organization_id": ge_cloud_organization_id, - }, - (cloud_base_url, cloud_access_token, cloud_organization_id, True), - id="conflicting_args", - ), - # The interaction between cloud_mode and ge_cloud_mode is very particular so we take extra care testing here: - pytest.param( - { - "cloud_mode": True, - "ge_cloud_mode": False, - }, - (None, None, None, True), - id="cloud_mode=True_overrides_ge_cloud_mode", - ), - pytest.param( - { - "cloud_mode": False, - "ge_cloud_mode": True, - }, - (None, None, None, False), - id="cloud_mode=False_overrides_ge_cloud_mode", - ), - pytest.param( - { - "cloud_mode": None, - "ge_cloud_mode": False, - }, - (None, None, None, False), - id="ge_cloud_mode=False_overrides_cloud_mode=None", - ), - pytest.param( - { - "cloud_mode": None, - "ge_cloud_mode": True, - }, - (None, None, None, True), - id="ge_cloud_mode=True_overrides_cloud_mode=None", - ), - ], -) -def test_get_context_resolve_cloud_args( - cloud_args: dict, expected_resolved_args: tuple -): - actual_resolved_args = get_context_resolver(**cloud_args) - assert actual_resolved_args == expected_resolved_args - - @pytest.mark.cloud @pytest.mark.parametrize( "id, ge_cloud_id, expected", diff --git a/tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py b/tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py index f3d43a93137c..ec939d097bf3 100644 --- a/tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py +++ b/tests/integration/docusaurus/connecting_to_your_data/fluent_datasources/how_to_instantiate_a_specific_filesystem_data_context.py @@ -5,16 +5,18 @@ ``` """ +import great_expectations as gx +from great_expectations.data_context.data_context.file_data_context import ( + FileDataContext, +) import pathlib from great_expectations.data_context.data_context.file_data_context import ( FileDataContext, ) -import great_expectations as gx - # Python -# -path_to_context_root_folder = "/my_gx_project/" +# +path_to_project_root = "./my_project/" # project_root_dir = pathlib.Path.cwd().absolute() @@ -23,11 +25,13 @@ assert context assert context.root_directory == str(path_to_context_root_folder) - # Python # -context = gx.get_context(context_root_dir=path_to_context_root_folder) +context = gx.get_context(project_root_dir=path_to_project_root) # assert context -assert context.root_directory == str(path_to_context_root_folder) + +context_root_dir = pathlib.Path(context.root_directory) +assert context_root_dir.stem == FileDataContext.GX_DIR +assert context_root_dir.parent.stem == "my_project"