Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge template pyproject.toml into one #2926

Merged
merged 9 commits into from
Oct 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
* Removed deprecated `kedro.extras.ColorHandler`.
* The Kedro IPython extension is no longer available as `%load_ext kedro.extras.extensions.ipython`; use `%load_ext kedro.ipython` instead.
* Anonymous nodes are given default names of the form `<function_name>([in1;in2;...]) -> [out1;out2;...]`, with the names of inputs and outputs separated by semicolons.
* The default project template now has one `pyproject.toml` at the root of the project (containing both the packaging metadata and the Kedro build config).
* The `requirements.txt` in the default project template moved to the root of the project as well (hence dependencies are now installed with `pip install -r requirements.txt` instead of `pip install -r src/requirements.txt`).

## Migration guide from Kedro 0.18.* to 0.19.*
### DataSets
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/airflow_astronomer.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ To follow this tutorial, ensure you have the following:
pip install kedro-airflow~=0.4
```

5. Run `pip install -r src/requirements.txt` to install all dependencies.
5. Run `pip install -r requirements.txt` to install all dependencies.

### Deployment process

Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/aws_step_functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The rest of the tutorial will explain each step in the deployment process above

* Create a `conf/aws` directory in your Kedro project
* Put a `catalog.yml` file in this directory with the following content
* Ensure that you have `s3fs>=0.3.0,<0.5` defined in your `src/requirements.txt` so the data can be read from S3.
* Ensure that you have `s3fs>=0.3.0,<0.5` defined in your `requirements.txt` so the data can be read from S3.

<details>
<summary><b>Click to expand</b></summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ Open your newly-created notebook and create **four new cells** inside it. You wi
1. Before you import and run your Python code, you'll need to install your project's dependencies on the cluster attached to your notebook. Your project has a `requirements.txt` file for this purpose. Add the following code to the first new cell to install the dependencies:

```ipython
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/src/requirements.txt"
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/requirements.txt"
```

2. To run your project in your notebook, you must load the Kedro IPython extension. Add the following code to the second new cell to load the IPython extension:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ Create **four new cells** inside your notebook. You will fill these cells with c
1. Before you import and run your Python code, you'll need to install your project's dependencies on the cluster attached to your notebook. Your project has a `requirements.txt` file for this purpose. Add the following code to the first new cell to install the dependencies:

```ipython
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/src/requirements.txt"
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/requirements.txt"
```

2. To run your project in your notebook, you must load the Kedro IPython extension. Add the following code to the second new cell to load the IPython extension:
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/distributed.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ For better dependency management, we encourage you to containerise the entire pi
Firstly make sure your [project requirements are up-to-date](../kedro_project_setup/dependencies.md) by running:

```bash
pip-compile --output-file=<project_root>/src/requirements.txt --input-file=<project_root>/src/requirements.txt
pip-compile --output-file=<project_root>/requirements.txt --input-file=<project_root>/requirements.txt
```

We then recommend the [`Kedro-Docker`](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-docker) plugin to streamline the process of building the image. [Instructions for using this are in the plugin's README.md](https://github.com/kedro-org/kedro-plugins/blob/main/README.md).
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/single_machine.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ conda install -c conda-forge kedro
Install the project’s dependencies, by running the following in the project's root directory:

```console
pip install -r src/requirements.txt
pip install -r requirements.txt
```

After having installed your project on the remote server you can run the Kedro project as follows from the root of the project:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/development/commands_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,10 +273,10 @@ def run(

#### Install all package dependencies

The following runs [`pip`](https://github.com/pypa/pip) to install all package dependencies specified in `src/requirements.txt`:
The following runs [`pip`](https://github.com/pypa/pip) to install all package dependencies specified in `requirements.txt`:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

For further information, see the [documentation on installing project-specific dependencies](../kedro_project_setup/dependencies.md#install-project-specific-dependencies).
Expand Down
6 changes: 3 additions & 3 deletions docs/source/development/linting.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ type. [You can read more in the `isort` documentation](https://pycqa.github.io/i


### Install the tools
Install `black` and `ruff` by adding the following lines to your project's `src/requirements.txt`
Install `black` and `ruff` by adding the following lines to your project's `requirements.txt`
file:
```text
black # Used for formatting code
Expand All @@ -30,7 +30,7 @@ ruff # Used for linting, formatting and sorting module imports
To install all the project-specific dependencies, including the linting tools, navigate to the root directory of the
project and run:
```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```
Alternatively, you can individually install the linting tools using the following shell commands:
```bash
Expand Down Expand Up @@ -72,7 +72,7 @@ These hooks are run before committing your code to your repositories to automati
making code reviews easier and less time-consuming.

### Install `pre-commit`
You can install `pre-commit` along with other dependencies by including it in the `src/requirements.txt` file of your
You can install `pre-commit` along with other dependencies by including it in the `requirements.txt` file of your
Kedro project by adding the following line:
```text
pre-commit
Expand Down
6 changes: 3 additions & 3 deletions docs/source/experiment_tracking/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ cd spaceflights
Install the project's dependencies:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Set up the session store
Expand Down Expand Up @@ -268,7 +268,7 @@ In this section, we illustrate how to compare Matplotlib plots across experiment

### Update the dependencies

Update the `src/requirements.txt` file in your Kedro project by adding the following dataset to enable Matplotlib for your project:
Update the `requirements.txt` file in your Kedro project by adding the following dataset to enable Matplotlib for your project:

```text
kedro-datasets[matplotlib.MatplotlibWriter]~=1.1
Expand All @@ -278,7 +278,7 @@ seaborn~=0.12.1
And install the requirements with:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

### Add a plotting node
Expand Down
4 changes: 2 additions & 2 deletions docs/source/get_started/new_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ There are a few ways to create a new project once you have [set up Kedro](instal

Once you've created a project:

* You need to **navigate to its project folder** and **install its dependencies**: `pip install -r src/requirements.txt`
* You need to **navigate to its project folder** and **install its dependencies**: `pip install -r requirements.txt`
* **To run the project**: `kedro run`
* **To visualise the project**: `kedro viz`

Expand Down Expand Up @@ -79,7 +79,7 @@ kedro new --starter=pandas-iris
However you create a Kedro project, once `kedro new` has completed, the next step is to navigate to the project folder (`cd <project-name>`) and install dependencies with `pip` as follows:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

Now run the project:
Expand Down
14 changes: 7 additions & 7 deletions docs/source/kedro_project_setup/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,38 +5,38 @@
When you create a project, you then introduce additional dependencies for the tasks it performs.

## Project-specific dependencies
You can specify a project's exact dependencies in the `src/requirements.txt` file to make it easier for you and others to run your project in the future,
You can specify a project's exact dependencies in the `requirements.txt` file to make it easier for you and others to run your project in the future,

Check notice on line 8 in docs/source/kedro_project_setup/dependencies.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/kedro_project_setup/dependencies.md#L8

[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.
Raw output
{"message": "[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.", "location": {"path": "docs/source/kedro_project_setup/dependencies.md", "range": {"start": {"line": 8, "column": 1}}}, "severity": "INFO"}
and to avoid version conflicts downstream. This can be achieved with the help of [`pip-tools`](https://pypi.org/project/pip-tools/).
To install `pip-tools` in your virtual environment, run the following command:
```bash
pip install pip-tools
```

To add or remove dependencies to a project, edit the `src/requirements.txt` file, then run the following:
To add or remove dependencies to a project, edit the `requirements.txt` file, then run the following:

```bash
pip-compile <project_root>/src/requirements.txt --output-file <project_root>/src/requirements.lock
pip-compile <project_root>/requirements.txt --output-file <project_root>/requirements.lock
```

This will [pip compile](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) the requirements listed in
the `src/requirements.txt` file into a `src/requirements.lock` that specifies a list of pinned project dependencies
the `requirements.txt` file into a `requirements.lock` that specifies a list of pinned project dependencies
(those with a strict version). You can also use this command with additional CLI arguments such as `--generate-hashes`
to use `pip`'s Hash Checking Mode or `--upgrade-package` to update specific packages to the latest or specific versions.
[Check out the `pip-tools` documentation](https://pypi.org/project/pip-tools/) for more information.

```{note}
The `src/requirements.txt` file contains "source" requirements, while `src/requirements.lock` contains the compiled version of those and requires no manual updates.
The `requirements.txt` file contains "source" requirements, while `src/requirements.lock` contains the compiled version of those and requires no manual updates.
```

To further update the project requirements, modify the `src/requirements.txt` file (not `src/requirements.lock`) and re-run the `pip-compile` command above.
To further update the project requirements, modify the `requirements.txt` file (not `src/requirements.lock`) and re-run the `pip-compile` command above.

Check warning on line 31 in docs/source/kedro_project_setup/dependencies.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/kedro_project_setup/dependencies.md#L31

[Kedro.toowordy] 'modify' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'modify' is too wordy", "location": {"path": "docs/source/kedro_project_setup/dependencies.md", "range": {"start": {"line": 31, "column": 45}}}, "severity": "WARNING"}


## Install project-specific dependencies

To install the project-specific dependencies, navigate to the root directory of the project and run:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Workflow dependencies
Expand Down
2 changes: 1 addition & 1 deletion docs/source/nodes_and_pipelines/micro_packaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ You can package a micro-package by executing: `kedro micropkg package <micropkg_

`kedro micropkg package pipelines.data_processing`

* This will generate a new [source distribution](https://docs.python.org/3/distutils/sourcedist.html) for this micro-package.
* This will generate a new [source distribution](https://docs.python.org/3.11/distutils/sourcedist.html) for this micro-package.
* By default, the tar file will be saved into `dist/` directory inside your project.
* You can customise the target with the `--destination` (`-d`) option.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/tutorial/spaceflights_tutorial_faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ documentation on how to install relevant dependencies for kedro_datasets.pandas.
https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html
```

The Kedro Data Catalog is missing [dependencies needed to parse the data](../kedro_project_setup/dependencies.md#install-dependencies-related-to-the-data-catalog). Check that you have [all the project dependencies to `requirements.txt`](./tutorial_template.md#install-project-dependencies) and then call `pip install -r src/requirements.txt` to install them.
The Kedro Data Catalog is missing [dependencies needed to parse the data](../kedro_project_setup/dependencies.md#install-dependencies-related-to-the-data-catalog). Check that you have [all the project dependencies to `requirements.txt`](./tutorial_template.md#install-project-dependencies) and then call `pip install -r requirements.txt` to install them.

### Pipeline run

Expand Down
6 changes: 3 additions & 3 deletions docs/source/tutorial/tutorial_template.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Set up the spaceflights project

This section shows how to create a new project (with `kedro new` using the [Kedro spaceflights starter](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights)) and install project dependencies (with `pip install -r src/requirements.txt`).
This section shows how to create a new project (with `kedro new` using the [Kedro spaceflights starter](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights)) and install project dependencies (with `pip install -r requirements.txt`).

## Create a new project

Expand Down Expand Up @@ -28,7 +28,7 @@ cd spaceflights

Kedro projects have a `requirements.txt` file to specify their dependencies and enable sharable projects by ensuring consistency across Python packages and versions.

The spaceflights project dependencies are stored in `src/requirements.txt`(you may find that the versions differ slightly depending on the version of Kedro):
The spaceflights project dependencies are stored in `requirements.txt`(you may find that the versions differ slightly depending on the version of Kedro):

```text
# code quality packages
Expand Down Expand Up @@ -60,7 +60,7 @@ scikit-learn~=1.0
To install all the project-specific dependencies, run the following from the project root directory:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Optional: logging and configuration
Expand Down
2 changes: 1 addition & 1 deletion docs/source/visualisation/kedro-viz_visualisation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ When prompted for a project name, you can enter anything, but we will assume `Sp
When your project is ready, navigate to the root directory of the project and install the dependencies for the project, which include Kedro-Viz:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

The next step is optional, but useful to check that all is working. Run the full set of pipelines for the tutorial project:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/visualisation/visualise_charts_with_plotly.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ kedro-datasets[plotly.PlotlyDataSet, plotly.JSONDataSet]~=1.1
Navigate to the root directory of the project in your terminal and install the dependencies for the tutorial project:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

### Configure the Data Catalog
Expand Down Expand Up @@ -177,7 +177,7 @@ You can view Matplotlib charts in Kedro-Viz when you use the [Kedro MatplotLibWr

### Update the dependencies

You must update the `src/requirements.txt` file in the Kedro project by adding the following dataset to enable Matplotlib for the project:
You must update the `requirements.txt` file in the Kedro project by adding the following dataset to enable Matplotlib for the project:

```bash
kedro-datasets[matplotlib.MatplotlibWriter]~=1.1
Expand Down
4 changes: 1 addition & 3 deletions features/environment.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,7 @@ def _setup_minimal_env(context):

def _install_project_requirements(context):
install_reqs = (
Path(
"kedro/templates/project/{{ cookiecutter.repo_name }}/src/requirements.txt"
)
Path("kedro/templates/project/{{ cookiecutter.repo_name }}/requirements.txt")
.read_text(encoding="utf-8")
.splitlines()
)
Expand Down
10 changes: 5 additions & 5 deletions features/steps/cli_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ def create_config_file(context):
@given("I have installed the project dependencies")
def pip_install_dependencies(context):
"""Install project dependencies using pip."""
reqs_path = "src/requirements.txt"
reqs_path = "requirements.txt"
res = run(
[context.pip, "install", "-r", reqs_path],
env=context.env,
Expand Down Expand Up @@ -409,7 +409,7 @@ def update_pyproject_toml(context: behave.runner.Context, new_source_dir):
@given("I have updated kedro requirements")
def update_kedro_req(context: behave.runner.Context):
"""Remove kedro as a standalone requirement."""
reqs_path = context.root_project_dir / "src" / "requirements.txt"
reqs_path = context.root_project_dir / "requirements.txt"

if reqs_path.is_file():
old_reqs = reqs_path.read_text().splitlines()
Expand All @@ -427,7 +427,7 @@ def update_kedro_req(context: behave.runner.Context):

@when("I add {dependency} to the requirements")
def add_req(context: behave.runner.Context, dependency: str):
reqs_path = context.root_project_dir / "src" / "requirements.txt"
reqs_path = context.root_project_dir / "requirements.txt"
if reqs_path.is_file():
reqs_path.write_text(reqs_path.read_text() + "\n" + str(dependency) + "\n")

Expand Down Expand Up @@ -605,14 +605,14 @@ def check_docs_generated(context: behave.runner.Context):
@then("requirements should be generated")
def check_reqs_generated(context: behave.runner.Context):
"""Check that new project requirements are generated."""
reqs_path = context.root_project_dir / "src" / "requirements.lock"
reqs_path = context.root_project_dir / "requirements.lock"
assert reqs_path.is_file()
assert "This file is autogenerated by pip-compile" in reqs_path.read_text()


@then("{dependency} should be in the requirements")
def check_dependency_in_reqs(context: behave.runner.Context, dependency: str):
reqs_path = context.root_project_dir / "src" / "requirements.txt"
reqs_path = context.root_project_dir / "requirements.txt"
assert dependency in reqs_path.read_text()


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ In order to get the best out of the template:

## How to install dependencies

Declare any dependencies in `src/requirements.txt` for `pip` installation.
Declare any dependencies in `requirements.txt` for `pip` installation.

To install them, run:

```
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## How to run Kedro
Expand All @@ -46,7 +46,7 @@ To configure the coverage threshold, look at the `.coveragerc` file.

## Project dependencies

To see and update the dependency requirements for your project use `src/requirements.txt`. You can install the project requirements with `pip install -r src/requirements.txt`.
To see and update the dependency requirements for your project use `requirements.txt`. You can install the project requirements with `pip install -r requirements.txt`.

[Further information about project dependencies](https://docs.kedro.org/en/stable/kedro_project_setup/dependencies.html#project-specific-dependencies)

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,40 @@
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "{{ cookiecutter.python_package }}"
readme = "README.md"
dynamic = ["dependencies", "version"]
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"

[project.optional-dependencies]
docs = [
"docutils<0.18.0",
"sphinx~=3.4.3",
"sphinx_rtd_theme==0.5.1",
"nbsphinx==0.8.1",
"sphinx-autodoc-typehints==1.11.1",
"sphinx_copybutton==0.3.1",
"ipykernel>=5.3, <7.0",
"Jinja2<3.1.0",
"myst-parser~=0.17.2",
]

[tool.setuptools.dynamic]
dependencies = {file = "requirements.txt"}
version = {attr = "{{ cookiecutter.python_package }}.__version__"}

[tool.setuptools.packages.find]
where = ["src"]
namespaces = false

[tool.kedro]
project_name = "{{ cookiecutter.project_name }}"
project_version = "{{ cookiecutter.kedro_version }}"
package_name = "{{ cookiecutter.python_package }}"
project_name = "{{ cookiecutter.project_name }}"
kedro_init_version = "{{ cookiecutter.kedro_version }}"
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved

[tool.pytest.ini_options]
addopts = """
Expand Down
Loading