Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move ATM fraud to examples repo #425

Merged
merged 16 commits into from
Jan 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ jobs:
os:
- ubuntu-22.04
- windows-2022
python-version: [ "3.10", "3.11" ]
python-version: ["3.10", "3.11"]
runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
with:
submodules: true

- name: Install Python and set up Poetry
uses: bakdata/ci-templates/actions/python-setup-poetry@v1.5.3
Expand Down Expand Up @@ -69,7 +71,7 @@ jobs:

- name: Install docs dependencies
run: poetry install --with docs

- name: Check markdown, toml, css formatting
uses: dprint/check@v2.2
if: runner.os == 'Linux'
Expand All @@ -83,21 +85,21 @@ jobs:
uses: bakdata/ci-templates/.github/workflows/python-poetry-publish-snapshot.yaml@v1.25.0
secrets:
pypi-token: ${{ secrets.TEST_PYPI_TOKEN }}

publish-docs-from-main:
runs-on: ubuntu-22.04
if: ${{ github.ref == 'refs/heads/main' }}
needs: [test]
steps:
- uses: actions/checkout@v3

- name: Publish docs from main branch
uses: ./.github/actions/update-docs
with:
username: ${{ secrets.GH_USERNAME }}
email: ${{ secrets.GH_EMAIL }}
token: ${{ secrets.GH_TOKEN }}
version: main
version: main

publish-dev-docs-from-pr:
runs-on: ubuntu-22.04
Expand All @@ -118,11 +120,11 @@ jobs:
docs:
- added|deleted|modified: 'docs/**'

- name: Publish dev docs from PR
- name: Publish dev docs from PR
if: steps.docs-changes.outputs.docs == 'true'
uses: ./.github/actions/update-docs
with:
username: ${{ secrets.GH_USERNAME }}
email: ${{ secrets.GH_EMAIL }}
token: ${{ secrets.GH_TOKEN }}
version: dev
version: dev
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "examples"]
path = examples
url = https://github.com/bakdata/kpops-examples
17 changes: 3 additions & 14 deletions docs/docs/resources/examples/defaults.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,15 @@
# Example `defaults.yaml` files

## [ATM Fraud Pipeline](https://github.com/bakdata/kpops/tree/main/examples/bakdata/atm-fraud-detection){target=_blank}
## [KPOps examples](https://github.com/bakdata/kpops-examples){target=_blank}

<!-- dprint-ignore-start -->

??? example "defaults.yaml"
```yaml
--8<--
https://raw.githubusercontent.com/bakdata/kpops/main/examples/bakdata/atm-fraud-detection/defaults.yaml
--8<--
```

<!-- dprint-ignore-end -->

## [Word-count Pipeline](https://github.com/bakdata/kpops-examples/tree/main/word-count){target=_blank}
Shared defaults for [ATM Fraud Pipeline](https://github.com/bakdata/kpops-examples/tree/main/atm-fraud){target=_blank} and [Word-count Pipeline](https://github.com/bakdata/kpops-examples/tree/main/word-count){target=_blank}

<!-- dprint-ignore-start -->

??? example "defaults.yaml"
```yaml
--8<--
https://raw.githubusercontent.com/bakdata/kpops-examples/main/word-count/defaults.yaml
https://raw.githubusercontent.com/bakdata/kpops-examples/main/defaults.yaml
--8<--
```

Expand Down
27 changes: 12 additions & 15 deletions docs/docs/user/examples/atm-fraud-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ postgresql bitnami/postgresql
??? example "PostgreSQL Example Helm chart values (`postgresql.yaml`)"
```yaml
auth:
database: app_db
database: app_db
enablePostgresUser: true
password: AppPassword
postgresPassword: StrongPassword
Expand Down Expand Up @@ -74,20 +74,21 @@ kubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:808

<!-- dprint-ignore-start -->

1. Export environment variables in your terminal:
1. Clone the [kpops-examples repository](https://github.com/bakdata/kpops-examples){target=_blank} and `cd` into the directory.

2. Install KPOps `pip install -r requirements.txt`.

3. Export environment variables in your terminal:

```shell
export DOCKER_REGISTRY=bakdata && \
export NAMESPACE=kpops
```

2. Deploy the pipeline
4. Deploy the pipeline

```shell
poetry run kpops deploy ./examples/bakdata/atm-fraud-detection/pipeline.yaml \
--pipeline-base-dir ./examples \
--config ./examples/bakdata/atm-fraud-detection/config.yaml \
--execute
kpops deploy atm-fraud/pipeline.yaml --execute
```

!!! Note
Expand Down Expand Up @@ -116,7 +117,7 @@ You should be able to see pipeline shown in the image below:
<!-- dprint-ignore-start -->

!!! Attention
Kafka Connect needs some time to set up the connector.
Kafka Connect needs some time to set up the connector.
Moreover, Streams Explorer needs a while to scrape the information from Kafka connect.
Therefore, it might take a bit until you see the whole graph.

Expand Down Expand Up @@ -146,11 +147,7 @@ helm --namespace kpops uninstall postgresql
2. Remove the pipeline

```shell
poetry run kpops clean ./examples/bakdata/atm-fraud-detection/pipeline.yaml \
--pipeline-base-dir ./examples \
--config ./examples/bakdata/atm-fraud-detection/config.yaml \
--verbose \
--execute
kpops clean atm-fraud/pipeline.yaml --verbose --execute
```

!!! Note
Expand All @@ -166,12 +163,12 @@ helm --namespace kpops uninstall postgresql

- `deploy` fails:
1. Read the error message.
2. Try to correct the mistakes if there were any. Likely the configuration is not correct or the port-forwarding is not working as intended.
2. Try to correct the mistakes if there were any. Likely the configuration is incorrect, or the port-forwarding is not working as intended.
3. Run `clean`.
4. Run `deploy --dry-run` to avoid havig to `clean` again. If an error is dropped, start over from step 1.
5. If the dry-run is succesful, run `deploy`.
- `clean` fails:
1. Read the error message.
2. Try to correct the indicated mistakes if there were any. Likely the configuration is not correct or the port-forwarding is not working as intended.
2. Try to correct the indicated mistakes if there were any. Likely the configuration is incorrect, or the port-forwarding is not working as intended.
3. Run `clean`.
4. If `clean` fails, follow the steps in [teardown](../getting-started/teardown.md).
9 changes: 2 additions & 7 deletions docs/docs/user/getting-started/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,7 @@ kubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:808
4. Deploy the pipeline

```shell
kpops deploy word-count/pipeline.yaml \
--defaults word-count/defaults.yaml \
--execute
kpops deploy word-count/pipeline.yaml --execute
```

!!! Note
Expand Down Expand Up @@ -145,10 +143,7 @@ helm --namespace kpops uninstall redis
2. Remove the pipeline

```shell
kpops clean word-count/pipeline.yaml \
--defaults word-count/defaults.yaml \
--verbose \
--execute
kpops clean word-count/pipeline.yaml --verbose --execute
```

!!! Note
Expand Down
1 change: 1 addition & 0 deletions examples
disrupted marked this conversation as resolved.
Show resolved Hide resolved
Submodule examples added at f76134
19 changes: 0 additions & 19 deletions examples/bakdata/atm-fraud-detection/config.yaml

This file was deleted.

32 changes: 0 additions & 32 deletions examples/bakdata/atm-fraud-detection/defaults.yaml

This file was deleted.

104 changes: 0 additions & 104 deletions examples/bakdata/atm-fraud-detection/pipeline.yaml

This file was deleted.

Loading
Loading