Fix/s3 publshing #12

This adds a new configuration setting: AzurePublishEndpoints, similar to the existing S3PublishEndpoints and SwiftPublishEndpoints. For each endpoint, the following has to be defined: - accountName - accountKey - container - prefix Azure tests require the following environment variables to be set: - AZURE_STORAGE_ACCOUNT - AZURE_STORAGE_ACCESS_KEY With either of these not set, Azure-specific tests are skipped.

Co-authored-by: Andrej Shadura <andrew@shadura.me>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

this is needed so concurrent reads and writes are possible.

This way db usage is safe.

Progress is not safe so for api its always nil and code needs to take care of this

This should avoid deadlocking when context is destroyed.

each key needs to be provided as separate argument to gpg1 --recv-keys

Fix #1

For read-only action transactions are not necessary and they risk to deadlock if multiple go-routines try to read the database.

This way, if no pre-request flag is specified, the globally configured default is used.

So they can run on e.g. LXC containers as root, or other conceivable setups.

…o the Travis CI Ubuntu 16.04 environment.

…y used Travis CI setup.

… workflow.

PublishRepo26Test fails to run because something in the CI environment forces gpg to ask for the user's password. Try to require gpg1 for the test, which seems to run fine in other environments.

See #1012

It may happen that aptly retries to download data during tests (maybe because of a network issue), but our fixtures doesn't account for it. So, we strip those irrelevant lines before comparison.

Fix #938

"temp" downloader uses its own naming for downloaded files.

Also, update "pkg/errors" library.

We use go modules now.

dpkg-gencontrol can be called with -v flag which set binary package's version separated from source version. When this happen, the Source field will contain version number in addition to source package name. This tripped Aptly's dbgsym restriction, which check for exact source package name, which in turn prevents the dbgsym & the whole .changes file from being imported. From the git history, it seems like this condition is a leftover from when Aptly filter dbgsym packages using "*-dbgsym". So, I decided to remove it. A test case has been added to prevent regression.

Fix #1034

Fix #1032

…tation

Aptly allows create e.g. repos with a / to use those with the REST api the router needs to allow urlencoded parameters in various places to represent this. A specific example of this is the /api/repos/:name/packages path Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Most modern distribution use python3 for python (3). Default to that to make it a bit simpler to run systems tests on Debian Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Seems go.mod had some modules that are no longer used since the last version bumps? Running `make modules` or really `go mod tidy` automagically cleans those up. Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

golangs compress/gzip isn't a parallel implementation, so it's quite a bit slower on most modern servers then pgzip. The below benchmark run shows that publishing a debian bullseye mirror snapshot (amd64, arm64, armhf, source) shows a gain of about 35% in publishing time (when skipping bz2 using MR #1081) ``` hyperfine -w 1 -m 3 -L aptly aptly-nobz2,aptly-nobz2-pgzip -p "{aptly} -config aptly.conf publish drop bullseye || true" "{aptly} -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye" Benchmark 1: aptly-nobz2 -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye Time (mean ± σ): 35.548 s ± 0.378 s [User: 39.465 s, System: 10.046 s] Range (min … max): 35.149 s … 35.902 s 3 runs Benchmark 2: aptly-nobz2-pgzip -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye Time (mean ± σ): 26.592 s ± 0.069 s [User: 42.207 s, System: 9.676 s] Range (min … max): 26.521 s … 26.660 s 3 runs Summary 'aptly-nobz2-pgzip -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye' ran 1.34 ± 0.01 times faster than 'aptly-nobz2 -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye' ``` Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Using bzip2 generates smaller index files (roughly 20% smaller Packages files) but it comes with a big performance penalty. When publishing a debian mirror snapshot (amd64, arm64, armhf, source) without contents skipping bzip speeds things up around 1.8 times. ``` $ hyperfine -w 1 -L skip-bz2 true,false -m 3 -p "aptly -config aptly.conf publish drop bullseye || true" "aptly -config aptly.conf publish snapshot --skip-bz2={skip-bz2} --skip-contents --skip-signing bullseye" Benchmark 1: aptly -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye Time (mean ± σ): 35.567 s ± 0.307 s [User: 39.366 s, System: 10.075 s] Range (min … max): 35.311 s … 35.907 s 3 runs Benchmark 2: aptly -config aptly.conf publish snapshot --skip-bz2=false --skip-contents --skip-signing bullseye Time (mean ± σ): 64.740 s ± 0.135 s [User: 68.565 s, System: 10.129 s] Range (min … max): 64.596 s … 64.862 s 3 runs Summary 'aptly -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye' ran 1.82 ± 0.02 times faster than 'aptly -config aptly.conf publish snapshot --skip-bz2=false --skip-contents --skip-signing bullseye' ``` Allow skipping bz2 creation for setups where faster publishing is more important then Package file size. Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

This change lets you disable ACL when using S3 by using a configuration value of `none`. This way we maintain backward compatibility with the default setting being `private`. Fixes: #1067

Add the https redirect to the gold ouptut of the test as this was changed for the jenkins debian repos and the tests were failing after this change.

To capture the coverage also for the integration tests, a test only executing the cmd.Run function is used. The test always exits with code 0 and prints the real exit code to stdout. Otherwise no coverage report is generated. Those changes enable a more accurate coverage report for future contributions.

The ".../packages" endpoints for mirror, local repos and snapshots all share the same syntax for querying. However the "/api/packages" endpoint doesn't match this. Adjust that to allow for a bit more consistency and allow querying the full package database. The current endpoint functionality "/packages/:name" is kept intact and can be used the same as now Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

While testing out Aptly, the `apt-get` client complains with the following error, since the `codename` was switched from the InRelease files that are baked out by Aptly: ``` E: Repository 'http://debianrepo.example.com/bionic testing InRelease' changed its 'Codename' value from '' to 'testing' ```

…ted style

All other api endpoints also send empty arrays instead of nil. Closes #1123

In v1.4.0 it [returned the snapshot](https://github.com/aptly-dev/aptly/blob/v1.4.0/api/snapshot.go#L168), but this was removed (by accident) in v1.5.0. This adds it back.

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.25.0 to 1.33.0. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/v1.33.0/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.25.0...v1.33.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

… type correctly

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.33.0 to 1.34.0. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/v1.34.0/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.33.0...v1.34.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

If aptly.EnableDebug is active, we use Debug, otherwise we use gin.ReleaseMode to remove the annoying nuding messages when running the api. fixes: #1103

Deleting a publish that does not exist now results in a status code 404 instead of 500. Fixes: #1006

The Pipeline will only fail if the code coverage has fallen more than 2 Percent. fixes: #1154

fixes: #1160

Replaced 2 with TASK_SUCCEEDED, 3 with TASK_FAILED. fixes: #1158

Is part of Issue #176

refs #1017 #975

…usly

… metrics

This should fix some tests, as a lot of them are dependent on deb.debian.org which no longer supports Debian 9 "Stretch". Instead we use archive.debian.org which will continue to contain "Stretch" packages for a long time.

Updated urllib, requests and requests_unixsocket

Older versions go-xz didn't wait for child processes meaning for exery unpack action a defunct xz would stick around. This got fixed in 0.1.0 Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

A race condition for publishing packages and mirrors at the same time was introduced in commit 77d7c38. The problem is that when opening a leveldb transaction and performing another 'put' to the db the system freezes.

When trying to delete a mirror that has snapshot and not providing the force option, the API should not return a `500 StatusInternalServerError`. A `403 StatusForbidden` is more appropriate when the condition is expected by the server.

Fixes: #1106

Updated the repo key, repo links in tests (jessie-cran35 -> bullseye-cran40) and the expected test output. Fixes: #1218

The current regex runs in exponential time, which massively impacts the runtime of the test suite, taking several seconds (~4s on my system) just to perform a single match. By replacing the mix of re.findall + the initial capture group with re.search + some string slicing, the time spent matching the regex becomes nearly instant, e.g.: $ make system-test TESTS='Config*' goes from taking ~10s to ~1.5s. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

1.20 changes the output format of coverage checks slightly to include a package name on each line, followed by `coverage:`, but the current regex assumes that the line *starts* with `coverage:`. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This will properly close the db and, more particularly, flush out any profile files being written. Otherwise, they can end up empty or truncated. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.15.0 to 0.17.0. - [Commits](golang/net@v0.15.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

…crypto/openpgp`

as per contributor instructions

Commits on Feb 11, 2022

Add release to CI

randombenj committed Feb 11, 2022

Configuration menu

View commit details

Copy full SHA for 21029c3

Browse repository at this point

Copy the full SHA

21029c3 View commit details

Browse the repository at this point in the history

Commits on Apr 13, 2022

Fix artifacts publishing

randombenj committed Apr 13, 2022

Configuration menu

View commit details

Copy full SHA for d955b06

Browse repository at this point

Copy the full SHA

d955b06 View commit details

Browse the repository at this point in the history

Commits on Jun 9, 2022

Update azurite dir

chuan authored and randombenj committed Jun 9, 2022

Configuration menu

View commit details

Copy full SHA for f9f1c8e

Browse repository at this point

Copy the full SHA

f9f1c8e View commit details

Browse the repository at this point in the history

Fix/s3 publshing #12

Fix/s3 publshing #12

Commits on Jan 21, 2022

Commits on Jan 27, 2022

Commits on Jan 31, 2022

Commits on Feb 5, 2022

Commits on Feb 8, 2022

Commits on Feb 11, 2022

Commits on Feb 13, 2022

Commits on Mar 30, 2022

Commits on Apr 4, 2022

Commits on Apr 5, 2022

Commits on Apr 12, 2022

Commits on Apr 13, 2022

Commits on Apr 25, 2022

Commits on Apr 27, 2022

Commits on Apr 29, 2022

Commits on Jun 6, 2022

Commits on Jun 9, 2022

Commits on Jun 15, 2022

Commits on Jun 20, 2022

Commits on Jun 21, 2022

Commits on Jun 22, 2022

Commits on Jul 13, 2022

Commits on Aug 4, 2022

Commits on Aug 16, 2022

Commits on Aug 29, 2022

Commits on Sep 9, 2022

Commits on Nov 17, 2022

Commits on Dec 12, 2022

Commits on Dec 22, 2022

Commits on Jan 10, 2023

Commits on Jan 23, 2023

Commits on Feb 20, 2023

Commits on Mar 7, 2023

Commits on Mar 13, 2023

Commits on Mar 14, 2023

Commits on Mar 22, 2023

Commits on May 15, 2023

Commits on May 19, 2023

Commits on May 26, 2023

Commits on May 31, 2023

Commits on Jul 14, 2023

Commits on Aug 18, 2023

Commits on Sep 13, 2023

Commits on Sep 14, 2023

Commits on Sep 21, 2023

Commits on Oct 3, 2023

Commits on Oct 24, 2023

Commits on Nov 1, 2023

Commits on Nov 23, 2023

Commits on Jan 13, 2024