Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DPE-3062] Removing binary deps #179

Merged
merged 21 commits into from
Feb 5, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions charmcraft.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ parts:
- rustc
- cargo
- pkg-config
charm-binary-python-packages:
- psycopg2-binary
- psycopg-binary
- libpq-dev
libpq:
build-packages:
- libpq-dev
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will be installed by the other plugin but might as well.

plugin: dump
source: /usr/lib/
source-type: local
prime:
- lib/
organize:
"*-linux-gnu/libpq.so*": lib/
Comment on lines +38 to +39
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@delgod requested moving the SO to lib.

95 changes: 26 additions & 69 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 8 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,14 @@ poetry-core = "^1.8.1"
lightkube = "^0.15.0"
lightkube-models = "^1.29.0.6"
pydantic = "^1.10.14"
psycopg2-binary = "^2.9.9"
psycopg = {extras = ["binary"], version = "^3.1.17"}


[tool.poetry.group.cdeps]
optional = true

[tool.poetry.group.cdeps.dependencies]
psycopg2 = "^2.9.9"
psycopg = {extras = ["c"], version = "^3.1.17"}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If installing psycopg in the main group. It will be compiled at each unit test run and the unit tests won't be able to run on systems that do not have libpq installed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't this this group is necessary or should be added

Installing pyscopg2 without --no-binary (aka with poetry, during unit tests) works fine on Ubuntu

Installing pyscopg2 with --no-binary (aka with charmcraft pack), with the changes Dragomir made, works fine

So, it's possible to only have pyscopg2 and pyscopg in the main dependency group, remove psycopg2-binary from the unit dependency group, and remove the c dependency group entirely

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Libpq was not present for sure on the self-hosted runner images nor is it a given that unit tests would be run on Ubuntu flavour that has libpq. Using compiled psycopg also means that CI unit tests would have to recompile on each run. @marceloneppel @taurus-forever what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit tests would have to recompile on each run

wdym by recompile?

fwiw pip install pyscopg2 on ubuntu runs quite quickly (few seconds)—I believe it's building a wheel from source but it doesn't seem to be any slower than a normal pip install from a binary wheel download

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this version unit tests task passes in ~10s: https://github.com/canonical/pgbouncer-k8s-operator/actions/runs/7714293497/job/21026204594?pr=179
If using just psycopg2 from the main group unit tests task takes ~30s: https://github.com/canonical/pgbouncer-k8s-operator/actions/runs/7714361605/job/21026430075?pr=203

Considering that the difference is up to declaring dependencies differently, I think this is the better approach.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems like something weird is going on

poetry add pyscopg2 takes 8 seconds (including venv creation) on a fresh gh runner: https://github.com/canonical/pgbouncer-k8s-operator/actions/runs/7723418033/job/21053662267

that doesn't account for a >= 20 second difference

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing pre-compiled binary here has a way bigger strategic effect comparing to 20 seconds saving.
BTW, 20 seconds is a noticeable time, we can safe, but as a separate PR IMHO.

Please, branch this to a new bugreport and unblock this huge merge please (as both options are safe and does the job). Tnx!


[tool.poetry.group.charm-libs.dependencies]
# data_platform_libs/v0/data_interfaces.py
Expand Down
2 changes: 1 addition & 1 deletion src/charm.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/usr/bin/env python3
#!/usr/bin/env -S LD_LIBRARY_PATH=lib python3
# Copyright 2023 Canonical Ltd.
# See LICENSE file for licensing details.

Expand Down
2 changes: 1 addition & 1 deletion tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ allowlist_externals =
charmcraftcache
mv
commands_pre =
poetry export --only main,charm-libs --output requirements.txt
poetry export --only main,charm-libs,cdeps --output requirements.txt
commands =
build-production: charmcraft pack {posargs}
build-dev: charmcraftcache pack {posargs}
Expand Down
Loading