Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use correct parallel build count #509

Closed
wants to merge 57 commits into from

Conversation

mattculler
Copy link
Contributor

  • Have you followed the guidelines for contributing?
  • Have you signed the CLA?
  • Have you successfully run tox?

(CRAFT-3524)

lengau and others added 30 commits September 6, 2024 15:41
This sets the expected API that clients (the Application) will use:

- Creating a fetch-service is done in setup();
- Sessions are created with create_session(), which receives a managed
  instance because we'll need it to figure out the network gateway;
- Sessions are destroyed with teardown_session();
- Stopping the fetch-service is done with shutdown().
Add a set of utility functions so that FetchService.setup() will spawn a
fetch-service if necessary. These utility functions include a function to
get the service's (json) status, and a function to tell whether the service
is up.

Use the $SNAP_USER_COMMON location for the fetch-service snap as a base for
the 'config' and 'spool' subdirectories.
Session credentials are stored to be used (in the near future) when setting up
the http/https proxies. When closing the session, the report obtained from the
fetch-service is discarded (for now, until we figure out what to do with it).
This configuration happens when creating the session. Currently we:

- Install the local certificate (added to this repo)
- Configure pip, snapd and apt
- Refresh apt's package listings to ensure all installations are traceable
  by the fetch-service

This initial implementation *always* refreshes apt listings, even if the
project build won't use them (e.g. no stage- or build- packages). In the
future we'll be smarter about this. The implementation also only supports
LXD instances, just because we need to figure out the instance's gateway
address and only the LXD work for that has been done so far. In the future
we can support all instance types with a cleaner craft-providers API.
This variable points to the local-ca.crt self-signed certificate, effectively
telling requests to trust that certificate when negotiating https requests.
Recent versions of the fetch-service not longer come with a fixed, "built-in"
certificate. Instead, clients are expected to generate their own. To do this,
we use a location that is shared among all Craft applications - the key and
certificate are generated if not found, and are otherwise shared by all
instances of all applications.

Fixes #21
This lets cargo allow our self-signed certificate when making https requests
to its registry.

Fixes #20
Now that the fetch-service's bug is fixed we can revert our temporary
change
This is temporary - in the near future we'll use the in-memory report to
generate a 'proper' manifest, but for now this lets us at least have something
to inspect/validate.
The strict mode is not yet implemented fully, so until further notice we should
create permissive sessions.
The snap's current confinement does not let it access the user's home files,
so <home>/.local/... is inaccessible.
There seems to be a bug currently with creating consecutive sessions, so for
now let's always shut down the fetch-service after the managed run is done.
With an open_stream() the output shows up in a single, updating line in brief
mode. This output is useful because the call takes a while.
This implementation checks the presence of the fetch-service snap through the
hardcoded path (/snap/bin/fetch-service). If, in the future, we need a more
sophisticated approach we can copy craft-provider's logic for checking the
lxd snap through snapd's api.

Fixes #36
With this commit the fetch-service service is disabled by default; the way
to enable it is through a new "--use-fetch-service" command line parameter.

This new parameter is initiallly only available to the "pack" command, as
the cache-clearing that the service needs precludes the use of other
lifecycle commands, and the manifest that the application will create based
on the fetch-service's session report is only produced during packing.

Fixes #15
This commit redirects the output of the fetch-service to a file. Since we plan
to have the fetch-service outlive the application that spawned it, we need to
use bash to redirect the fetch-service's output to a file in a way that
persists after the application ends.

Fixes #51
Test is failing due to canonical/fetch-service#208;
once that's fixed we can revert this commit. Note that only one of the ports
is failing; the proxy port is still giving the expected error output.
Signed-off-by: Callahan Kovacs <callahankovacs@gmail.com>
chore: merge release 4.1.3 into main
This allows applications to override the inner run logic, with the end
goal of capturing application specific exceptions and raise
appropriate ones for Craft Application to handle.

Without the logic split, overriding run to capture exceptions is
virtually impossible as the run method holds a generic Exception
handler.

Signed-off-by: Sergio Schvezov <sergio.schvezov@canonical.com>
This new parameter to ProviderService.instance() defaults to False and, if
True, will cause the service to clean a pre-existing instance and create a
new one.
@mattculler mattculler self-assigned this Oct 7, 2024
@mattculler mattculler linked an issue Oct 7, 2024 that may be closed by this pull request
@mattculler mattculler marked this pull request as ready for review October 7, 2024 21:58
@mattculler mattculler requested review from mr-cal and a team October 7, 2024 21:59
pyproject.toml Outdated Show resolved Hide resolved
Co-authored-by: Alex Lowe <alex.lowe@canonical.com>
craft_application/application.py Show resolved Hide resolved
docs/reference/changelog.rst Outdated Show resolved Hide resolved
Copy link
Contributor

@mr-cal mr-cal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add an integration test to ensure this is working as expected?

@mattculler
Copy link
Contributor Author

Can you add an integration test to ensure this is working as expected?

Yeah good idea, I'll do that.

@mattculler mattculler requested a review from lengau October 8, 2024 16:28
@mr-cal
Copy link
Contributor

mr-cal commented Oct 8, 2024

Don't forget to target hotfix/4.2, thanks!

@mattculler mattculler changed the base branch from main to hotfix/4.2 October 8, 2024 21:28
@mattculler
Copy link
Contributor Author

Cherry-picked relevant commits into #517 when this one went bad after changing destination from main to hotfix.

@mattculler mattculler closed this Oct 8, 2024
@mattculler mattculler deleted the work/CRAFT-3524-fix-build-count branch October 8, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect value for $CRAFT_PARALLEL_BUILD_COUNT in project metadata
5 participants