Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test/system: Connect system tests to Meson #1062

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

test/system: Connect system tests to Meson #1062

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a44a81
src: Assign tests to test suites
HarryMichal May 30, 2022
20e9f6b
test/system: Connect system tests to Meson
HarryMichal May 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 0 additions & 28 deletions meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,8 @@ subid_dep = cc.find_library('subid', has_headers: ['shadow/subid.h'])
go = find_program('go')
go_md2man = find_program('go-md2man')

bats = find_program('bats', required: false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it really necessary to move them? Isn't it a lot more obvious to have all the dependencies listed in the top level meson.build? Otherwise, distributors will have to go through all the meson.build files, which most won't do.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not necessary. To me it made sense to have purely testing dependencies specified in the test subdirectory. I can move those back to the root build file.

codespell = find_program('codespell', required: false)
htpasswd = find_program('htpasswd', required: false)
openssl = find_program('openssl', required: false)
podman = find_program('podman', required: false)
shellcheck = find_program('shellcheck', required: false)
skopeo = find_program('skopeo', required: false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of Podman and Skopeo, they aren't just dependencies for the test suite, but also runtime dependencies for the main toolbox(1) binary. So moving them under test/system can be misleading.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving the podman binary is an oversight and I didn't realise at the time of writing that skopeo is now a runtime dependency. That wasn't the case in the past.


bashcompletionsdir = get_option('bash_completions_dir')
if bashcompletionsdir == ''
Expand Down Expand Up @@ -84,29 +79,6 @@ if shellcheck.found()
test('shellcheck toolbox (deprecated)', shellcheck, args: [toolbox_sh])
endif

install_subdir(
'test',
install_dir: get_option('datadir') / meson.project_name(),
exclude_files: [
'system/libs/bats-assert/.git',
'system/libs/bats-assert/.gitignore',
'system/libs/bats-assert/.travis.yml',
'system/libs/bats-assert/package.json',
'system/libs/bats-support/.git',
'system/libs/bats-support/.gitignore',
'system/libs/bats-support/.travis.yml',
'system/libs/bats-support/package.json'
],
exclude_directories: [
'system/libs/bats-assert/.git',
'system/libs/bats-assert/script',
'system/libs/bats-assert/test',
'system/libs/bats-support/.git',
'system/libs/bats-support/script',
'system/libs/bats-support/test'
]
)

subdir('data')
subdir('doc')
subdir('profile.d')
Expand Down
4 changes: 2 additions & 2 deletions playbooks/system-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@
- include_tasks: build.yaml

- name: Run system tests
command: bats --timing ./test/system
command: meson test --verbose --suite system
environment:
PODMAN: '/usr/bin/podman'
TOOLBOX: '/usr/local/bin/toolbox'
args:
chdir: '{{ zuul.project.src_dir }}'
chdir: '{{ zuul.project.src_dir }}/builddir'
9 changes: 7 additions & 2 deletions playbooks/unit-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,12 @@
tasks:
- include_tasks: build.yaml

- name: Test
command: meson test -C builddir --verbose
- name: Linting tests
command: meson test -C builddir --verbose --suite linting
args:
chdir: '{{ zuul.project.src_dir }}'

- name: Unit tests
command: meson test -C builddir --verbose --suite unit
args:
chdir: '{{ zuul.project.src_dir }}'
32 changes: 28 additions & 4 deletions src/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,33 @@ custom_target(
)

if shellcheck.found()
test('shellcheck src/go-build-wrapper', shellcheck, args: [go_build_wrapper_file])
test(
'shellcheck go-build-wrapper',
shellcheck,
args: [go_build_wrapper_file],
suite: 'linting'
)
endif

test('go fmt', meson_go_fmt_program, args: [meson.current_source_dir()])
test('go vet', go, args: ['vet', '-c', '3', './...'], workdir: meson.current_source_dir())
test('go test', go, args: ['test', '-vet', 'off', './...'], workdir: meson.current_source_dir())
test(
'go fmt',
meson_go_fmt_program,
args: [meson.current_source_dir()],
suite: 'linting'
)

test(
'go vet',
go,
args: ['vet', '-c', '3', './...'],
suite: 'linting',
workdir: meson.current_source_dir()
)

test(
'go test',
go,
args: ['test', '-vet', 'off', './...'],
suite: 'unit',
workdir: meson.current_source_dir()
)
17 changes: 14 additions & 3 deletions test/system/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,19 @@ by running `git submodule init` and `git submodule update`.

First, make sure you have all the dependencies installed.

- Enter the toolbox root folder
- Invoke command `bats ./test/system/` and the test suite should fire up
### Meson (recommended)

> This method is optimal when bulding from source

- Set up the project per instructions at https://containertoolbx.org/install
- Invoke command `meson test --suite system` and the test suite should fire up

### Manually (legacy)

> This method is optimal if you want to test an existing build of Toolbx

- Enter the `test/system` directory in the project root directory
- Invoke command `bats ./` and the test suite should fire up

Mocking of images is done automatically to prevent potential networking issues
and to speed up the cases.
Expand All @@ -41,7 +52,7 @@ By default the test suite uses the system versions of `podman`, `skopeo` and
If you have a `podman`, `skopeo` or `toolbox` installed in a nonstandard
location then you can use the `PODMAN`, `SKOPEO` and `TOOLBOX` environmental
variables to set the path to the binaries. So the command to invoke the test
suite could look something like this: `PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./test/system/`.
suite could look something like this: `PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./`.

When running the tests, make sure the `test suite: [job]` jobs are successful.
These jobs set up the whole environment and are a strict requirement for other
Expand Down
42 changes: 42 additions & 0 deletions test/system/libs/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
test_system_libs_dir = test_system_dir / 'libs'

test_system_libs_files = files(
'helpers.bash'
)

install_data(
install_dir: test_system_libs_dir,
sources: test_system_libs_files
)

install_subdir(
'bats-support',
install_dir: test_system_libs_dir,
exclude_files: [
'.git',
'.gitignore',
'.travis.yml',
'package.json'
],
exclude_directories: [
'.git',
'script',
'test'
]
)

install_subdir(
'bats-assert',
install_dir: test_system_libs_dir,
exclude_files: [
'.git',
'.gitignore',
'.travis.yml',
'package.json'
],
exclude_directories: [
'.git',
'script',
'test'
]
)
42 changes: 41 additions & 1 deletion test/system/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
test_system_dir = get_option('datadir') / meson.project_name() / 'test'

awk = find_program('awk', required: false)
bats = find_program('bats', required: false)
htpasswd = find_program('htpasswd', required: false)
openssl = find_program('openssl', required: false)
podman = find_program('podman', required: false)
skopeo = find_program('skopeo', required: false)

test_system = files(
'001-version.bats',
'002-help.bats',
Expand All @@ -15,9 +24,40 @@ test_system = files(
'210-ulimit.bats',
'211-dbus.bats',
'setup_suite.bash',
'libs/helpers.bash',
)

if shellcheck.found()
test('shellcheck test/system', shellcheck, args: [test_system])
endif

if fs.exists('/run/.containerenv')
message('System tests can not be run in a container.')
elif not awk.found() or not bats.found() or not htpasswd.found() or not openssl.found() or not podman.found() or not skopeo.found()
message('System tests require: awk, bats, htpasswd, openssl, podman and skopeo being installed.')
else
system_tests_env = environment()
system_tests_env.set('TOOLBOX', toolbox_go.full_path())

test(
'system tests',
bats,
args: [
'--formatter', 'tap',
'--timing',
'./'],
depends: toolbox_go,
env: system_tests_env,
is_parallel: false,
protocol: 'tap',
suite: 'system',
timeout: 3600,
workdir: meson.current_source_dir(),
)
endif

install_data(
install_dir: test_system_dir,
sources: [test_system, 'README.md'],
)

subdir('libs')
2 changes: 1 addition & 1 deletion test/system/setup_suite.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

missing_dependencies=false

if [ -f test/system/libs/bats-assert/load.bash ] && [ -f test/system/libs/bats-support/load.bash ]; then
if [ -f libs/bats-assert/load.bash ] && [ -f libs/bats-support/load.bash ]; then
load 'libs/helpers'
else
missing_dependencies=true
Expand Down
Loading