Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use correct parallel build count #509

Closed
wants to merge 57 commits into from
Closed
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
dc1928f
fix(models): coerce numbers to strings by default (#450)
lengau Sep 6, 2024
22909db
feat: add initial FetchService skeleton (#7)
tigarmo Jun 10, 2024
ff76bb5
feat: start the fetch-service (#8)
tigarmo Jun 13, 2024
a047ae4
feat: create/teardown fetch-service sessions (#9)
tigarmo Jun 14, 2024
bdb374c
feat: configure build instances for fetch-service (#11)
tigarmo Jun 17, 2024
6bcddf9
fix: set REQUESTS_CA_BUNDLE for the fetch-service env. (#16)
tigarmo Jul 4, 2024
8c1096d
feat: generate the fetch-service certificate
tigarmo Jul 23, 2024
5a369df
fix: set CARGO_HTTP_CAINFO for the fetch-service env
tigarmo Jul 25, 2024
ae025ad
chore(test): set correct deb mimetype back
tigarmo Jul 26, 2024
d521195
feat: dump the fetch-service report
tigarmo Jun 27, 2024
33595c0
feat(fetch): default to creating permissive sessions
tigarmo Jul 26, 2024
5827eeb
fix: use fetch-service's common dir for certificate
tigarmo Jul 29, 2024
b6caa9c
chore: log the fetch-service command line
tigarmo Jul 29, 2024
b7355b5
feat: always shutdown the fetch-service
tigarmo Jul 29, 2024
15b3c1d
chore: redirect output of 'apt update'
tigarmo Jul 29, 2024
9a6d450
style(lint): fix ruff 0.6.0 fetch-service errors
tigarmo Aug 28, 2024
a70bf86
fix: better error if the fetch-service is missing (#55)
tigarmo Aug 29, 2024
ef85eee
feat: add an argument to enable the fetch service (#53)
tigarmo Aug 30, 2024
c9ccccf
feat: proper logging for the fetch-service (#56)
tigarmo Sep 6, 2024
c8e923a
chore: disable port conflict test (#64)
tigarmo Sep 11, 2024
1ae919c
docs(changelog): add release notes for 4.1.3 (#459)
mr-cal Sep 12, 2024
5be3430
chore: merge branch 'main' into release-merge/4.1.3
lengau Sep 12, 2024
1216017
chore: merge release 4.1.3 into main (#460)
tigarmo Sep 13, 2024
a66b217
Merge branch 'main' into hotfix-merge/4.2.1
lengau Sep 13, 2024
2d7805a
chore: merge 4.2.1 to main
lengau Sep 13, 2024
ac73285
chore: factor run logic outside of try block
sergiusens Sep 13, 2024
41b699e
chare: merge branch 'main' into hotfix-merge/4.2.2
lengau Sep 13, 2024
8ac3f3d
chore: merge 4.2.2 to main
lengau Sep 13, 2024
d872bfd
ci: use the default version of lxd for integration tests (#461)
lengau Sep 13, 2024
ba3be78
feat(provider): add a way to clean existing instances
tigarmo Sep 13, 2024
e931a59
feat: wipe existing instances when using the fetch-service
tigarmo Sep 13, 2024
e546fae
feat: set the fetch-service to idle shutdown
tigarmo Sep 6, 2024
d448c44
feat(docs): update partitions docs (#475)
mattculler Sep 18, 2024
f2834d9
chore: merge branch 'origin/main' into release-merge/4.2.3
lengau Sep 18, 2024
616b3f0
chore: merge 4.2.3 to main
lengau Sep 18, 2024
575ac20
feat: models for the craft manifest (#473)
tigarmo Sep 19, 2024
e258642
feat: generate project manifest in managed runs
tigarmo Sep 17, 2024
60f3188
feat: create the final craft manifest
tigarmo Sep 19, 2024
e3e079f
chore(merge): 4.2.4 into merge/4.2.4
mr-cal Sep 20, 2024
cb8197e
chore(merge): 4.2.4 into main
lengau Sep 20, 2024
07e89de
feat: support "--platform" for "clean"
tigarmo Sep 19, 2024
c971a60
feat: set GOPROXY=direct in fetch-service runs (#489)
tigarmo Sep 24, 2024
d327e60
feat: allow strict/permissive fetch-service sessions (#490)
tigarmo Sep 25, 2024
63ff3d0
build(deps): loosen requests dependency (#491)
lengau Sep 26, 2024
0b10cf6
ci: add security scanning job (#494)
lengau Oct 1, 2024
327ff94
feat: mark the fetch-service integration as experimental (#496)
tigarmo Oct 1, 2024
3bd339c
feat: add craft-platforms compatibility methods (#495)
lengau Oct 1, 2024
8b2b8f3
fix(docs): fix docs for canonical-sphinx 0.2 (#498)
lengau Oct 3, 2024
8b46697
feat: move get_parallel_build_count and tests to util
mattculler Oct 7, 2024
1c3829a
feat: pass parallel build count to ProjectInfo
mattculler Oct 7, 2024
3fdc863
chore: autoformat
mattculler Oct 7, 2024
72ac884
docs: add to changelog
mattculler Oct 7, 2024
cb3ceef
Update pyproject.toml
mattculler Oct 7, 2024
23b8142
fix(docs): clarified shoddy changelog message
mattculler Oct 8, 2024
8567282
test: add integration test for parallel build count
mattculler Oct 8, 2024
90507ae
chore: autoformat
mattculler Oct 8, 2024
cd2c9a4
Merge branch 'hotfix/4.2' into work/CRAFT-3524-fix-build-count
mattculler Oct 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/security-scan.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: Security scan
on:
pull_request:
push:
branches:
- main
- hotfix/*
- work/secscan # For development

jobs:
python-scans:
name: Scan Python project
uses: canonical/starflow/.github/workflows/scan-python.yaml@main
with:
packages: python-apt-dev
13 changes: 7 additions & 6 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,6 @@ jobs:
3.10
3.12
cache: 'pip'
- name: Setup LXD
uses: canonical/setup-lxd@v0.1.1
with:
channel: latest/stable
- name: Configure environment
run: |
echo "::group::apt-get"
Expand Down Expand Up @@ -111,10 +107,12 @@ jobs:
cache: 'pip'
- name: Setup LXD
uses: canonical/setup-lxd@v0.1.1
with:
channel: latest/stable
- name: Configure environment
run: |
echo "::group::Begin snap install"
echo "Installing snaps in the background while running apt and pip..."
sudo snap install --no-wait --channel=beta fetch-service
echo "::endgroup::"
echo "::group::apt-get"
sudo apt update
sudo apt-get install -y libapt-pkg-dev
Expand All @@ -123,6 +121,9 @@ jobs:
python -m pip install tox
echo "::endgroup::"
mkdir -p results
echo "::group::Wait for snap to complete"
snap watch --last=install
echo "::endgroup::"
- name: Setup Tox environments
run: tox run -e integration-${{ matrix.python }} --notest
- name: Integration tests
Expand Down
34 changes: 32 additions & 2 deletions craft_application/application.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,11 @@ def __init__(
else:
self._work_dir = pathlib.Path.cwd()

# Whether the command execution should use the fetch-service
self._use_fetch_service = False
# The kind of sessions that the fetch-service service should create
self._fetch_service_policy = "strict"

@property
def app_config(self) -> dict[str, Any]:
"""Get the configuration passed to dispatcher.load_command().
Expand Down Expand Up @@ -238,6 +243,11 @@ def _configure_services(self, provider_name: str | None) -> None:
build_plan=self._build_plan,
provider_name=provider_name,
)
self.services.update_kwargs(
"fetch",
build_plan=self._build_plan,
session_policy=self._fetch_service_policy,
)

def _resolve_project_path(self, project_dir: pathlib.Path | None) -> pathlib.Path:
"""Find the project file for the current project.
Expand Down Expand Up @@ -368,8 +378,14 @@ def run_managed(self, platform: str | None, build_for: str | None) -> None:
instance_path = pathlib.PosixPath("/root/project")

with self.services.provider.instance(
build_info, work_dir=self._work_dir
build_info,
work_dir=self._work_dir,
clean_existing=self._use_fetch_service,
) as instance:
if self._use_fetch_service:
session_env = self.services.fetch.create_session(instance)
env.update(session_env)

cmd = [self.app.name, *sys.argv[1:]]
craft_cli.emit.debug(
f"Executing {cmd} in instance location {instance_path} with {extra_args}."
Expand All @@ -387,6 +403,12 @@ def run_managed(self, platform: str | None, build_for: str | None) -> None:
raise craft_providers.ProviderError(
f"Failed to execute {self.app.name} in instance."
) from exc
finally:
if self._use_fetch_service:
self.services.fetch.teardown_session()

if self._use_fetch_service:
self.services.fetch.shutdown(force=True)

def configure(self, global_args: dict[str, Any]) -> None:
"""Configure the application using any global arguments."""
Expand Down Expand Up @@ -482,12 +504,14 @@ def _pre_run(self, dispatcher: craft_cli.Dispatcher) -> None:
At the time this is run, the command is loaded in the dispatcher, but
the project has not yet been loaded.
"""
args = dispatcher.parsed_args()

# Some commands might have a project_dir parameter. Those commands and
# only those commands should get a project directory, but only when
# not managed.
if self.is_managed():
self.project_dir = pathlib.Path("/root/project")
elif project_dir := getattr(dispatcher.parsed_args(), "project_dir", None):
elif project_dir := getattr(args, "project_dir", None):
self.project_dir = pathlib.Path(project_dir).expanduser().resolve()
if self.project_dir.exists() and not self.project_dir.is_dir():
raise errors.ProjectFileMissingError(
Expand All @@ -496,6 +520,11 @@ def _pre_run(self, dispatcher: craft_cli.Dispatcher) -> None:
resolution="Ensure the path entered is correct.",
)

fetch_service_policy: str | None = getattr(args, "fetch_service_policy", None)
if fetch_service_policy:
self._use_fetch_service = True
self._fetch_service_policy = fetch_service_policy

def get_arg_or_config(
self, parsed_args: argparse.Namespace, item: str
) -> Any: # noqa: ANN401
Expand Down Expand Up @@ -682,6 +711,7 @@ def _expand_environment(self, yaml_data: dict[str, Any], build_for: str) -> None
application_name=self.app.name, # not used in environment expansion
cache_dir=pathlib.Path(), # not used in environment expansion
arch=build_for_arch,
parallel_build_count=util.get_parallel_build_count(self.app.name),
mattculler marked this conversation as resolved.
Show resolved Hide resolved
project_name=yaml_data.get("name", ""),
project_dirs=project_dirs,
project_vars=environment_vars,
Expand Down
23 changes: 19 additions & 4 deletions craft_application/commands/lifecycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,19 @@
"""Basic lifecycle commands for a Craft Application."""
from __future__ import annotations

import argparse
import os
import pathlib
import subprocess
import textwrap
from typing import TYPE_CHECKING, Any
from typing import Any

from craft_cli import CommandGroup, emit
from craft_parts.features import Features
from typing_extensions import override

from craft_application.commands import base

if TYPE_CHECKING: # pragma: no cover
import argparse


def get_lifecycle_command_group() -> CommandGroup:
"""Return the lifecycle related command group."""
Expand Down Expand Up @@ -355,6 +353,14 @@ def _fill_parser(self, parser: argparse.ArgumentParser) -> None:
help="Output directory for created packages.",
)

parser.add_argument(
"--use-fetch-service",
help=argparse.SUPPRESS,
choices=("strict", "permissive"),
metavar="policy",
dest="fetch_service_policy",
)

@override
def _run(
self,
Expand Down Expand Up @@ -392,6 +398,9 @@ def _run(
_launch_shell()
raise

if parsed_args.fetch_service_policy and packages:
self._services.fetch.create_project_manifest(packages)

if not packages:
emit.progress("No packages created.", permanent=True)
elif len(packages) == 1:
Expand Down Expand Up @@ -427,6 +436,12 @@ def _fill_parser(self, parser: argparse.ArgumentParser) -> None:
nargs="*",
help="Optional list of parts to process",
)
parser.add_argument(
"--platform",
type=str,
metavar="name",
help="Platform to clean",
)

@override
def _run(
Expand Down
4 changes: 4 additions & 0 deletions craft_application/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -242,3 +242,7 @@ def __init__( # (too many arguments)
reportable=reportable,
retcode=retcode,
)


class FetchServiceError(CraftError):
"""Errors related to the fetch-service."""
Loading
Loading