From 74d464046ce6781d846035de0d9ef7a085b9cba6 Mon Sep 17 00:00:00 2001 From: a-dubs Date: Thu, 25 Jul 2024 17:06:49 -0400 Subject: [PATCH 1/4] doc: improve integrations testing configuration instructions --- doc/rtd/development/integration_tests.rst | 220 +++++++++++++++++++++- 1 file changed, 215 insertions(+), 5 deletions(-) diff --git a/doc/rtd/development/integration_tests.rst b/doc/rtd/development/integration_tests.rst index 5fe5845dd4b..7b1055797da 100644 --- a/doc/rtd/development/integration_tests.rst +++ b/doc/rtd/development/integration_tests.rst @@ -31,26 +31,179 @@ integration tests. To run all integration tests, you would run: .. code-block:: bash - $ tox -e integration-tests + tox -e integration-tests -``pytest`` arguments may also be passed. For example: +``pytest`` arguments may also be passed. + +Or, specify a path to run all tests inside the given file or directory: + +.. code-block:: bash + + tox -e integration-tests tests/integration_tests/modules/test_combined.py + + +.. code-block:: bash + + tox -e integration-tests tests/integration_tests/modules + +Or, run a specific test: .. code-block:: bash - $ tox -e integration-tests tests/integration_tests/modules/test_combined.py + tox -e integration-tests tests/integration_tests/modules/test_combined.py::test_bootcmd + +TODO: Maybe make these 3 commands in one tab group?? + Configuration ============= All possible configuration values are defined in -`tests/integration_tests/integration_settings.py`_. Defaults can be overridden +`tests/integration_tests/integration_settings.py`_. Look in this file for +the full list of variables that are available and for context on what each +variable does and what the default values are. by supplying values in :file:`tests/integration_tests/user_settings.py` or by providing an environment variable of the same name prepended with ``CLOUD_INIT_``. For example, to set the ``PLATFORM`` setting: .. code-block:: bash - CLOUD_INIT_PLATFORM='ec2' pytest tests/integration_tests/ + CLOUD_INIT_PLATFORM='ec2' tox -e integration_tests -- tests/integration_tests/ + + +Common integration test run configurations +========================================== + + +Keep instance after test run +------------------------------- + +By default, the test instance is torn down after the test run. To keep +the instance running after the test run, set the ``KEEP_INSTANCE`` variable +to ``True``. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_KEEP_INSTANCE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + KEEP_INSTANCE = True + + +Use in-place cloud-init source code +------------------------------------- + +The simplest way to test an integraton test using your current cloud-init +changes is to set the ``CLOUD_INIT_SOURCE`` to ``IN_PLACE``. This works ONLY +on LXD containers. This will mount the source code as-is directly into +the container to override the pre-existing cloudinit module. This +won't work for non-local LXD remotes and won't run any installation code. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_CLOUD_INIT_SOURCE=IN_PLACE tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + CLOUD_INIT_SOURCE = 'IN_PLACE' + + +Collecting logs after test run +------------------------------- + +By default, logs are collected only when a test fails, by running ``cloud-init +collect-logs`` on the instance. To collect logs after every test run, set the +``COLLECT_LOGS`` variable to ``ALWAYS``. + +By default, the logs are collected to the ``/tmp/cloud_init_test_logs`` +directory. To change the directory, set the ``LOCAL_LOG_PATH`` variable to +the desired path. + +TODO: can the collect log path be set to a relative path? + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_COLLECT_LOGS=ALWAYS CLOUD_INIT_LOCAL_LOG_PATH=/tmp/your-local-directory tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + COLLECT_LOGS = "ALWAYS" + LOCAL_LOG_PATH = "/tmp/logs" + + +Advanced test reporting and profiling +------------------------------------- + +For advanced test reporting, set the ``INCLUDE_COVERAGE`` variable to ``True``. +This will generate a coverage report for the integration test run, and the +report will be stored in an ``html`` directory inside the directory specified +by ``LOCAL_LOG_PATH``. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_INCLUDE_COVERAGE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + INCLUDE_COVERAGE = True + + +Addtionally, for profiling the integration tests, set the ``INCLUDE_PROFILE`` +variable to ``True``. This will generate a profile report for the integration +test run, and the report will be stored in the directory specified by +``LOCAL_LOG_PATH``. +# We default our coverage to False because it involves modifying the +# cloud-init systemd services, which is too intrusive of a change to +# enable by default. If changed to true, the test directory corresponding +# to the test run under LOCAL_LOG_PATH defined above will contain an +# `html` directory with the coverage report. +INCLUDE_COVERAGE = False + +# We default our profile to False because it involves modifying the +# cloud-init systemd services, which is too intrusive of a change to +# enable by default. If changed to true, the test directory corresponding +# to the test run under LOCAL_LOG_PATH defined above will contain a report +INCLUDE_PROFILE = False + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_INCLUDE_PROFILE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + INCLUDE_PROFILE = True + Cloud interaction ================= @@ -65,6 +218,39 @@ For a minimal setup using LXD, write the following to [lxd] + +For more information on configuring pycloudlib, see the +`pycloudlib configuration documentation`_. + +To specify a specific cloud to test against, first, ensure that your pycloudlib +configuration is set up correctly. Then, modify the ``PLATFORM`` variable to be +on of: + +- ``azure``: Microsoft Azure +- ``ec2``: Amazon EC2 +- ``gce``: Google Compute Engine +- ``ibm``: IBM Cloud +- ``lxd_container``: LXD container +- ``lxd_vm``: LXD VM +- ``oci``: Oracle Cloud Infrastructure +- ``openstack``: OpenStack +- ``qemu``: QEMU + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_PLATFORM='lxd_container' tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + PLATFORM = 'lxd_container' + + Image selection =============== @@ -108,6 +294,29 @@ via fixture. Image setup roughly follows these steps: * Take a snapshot of the instance to be used as a new image from which new instances can be launched. + +Keep image after test run +-------------------------- + +By default, the image created during the test run is torn down after +the test run. If further debugging is needed, you can keep the image snapshot +for further use by setting the ``KEEP_IMAGE`` variable to ``True``. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_KEEP_IMAGE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + KEEP_IMAGE = True + + Test setup ========== @@ -213,3 +422,4 @@ Customizing the launch arguments before launching an instance manually: .. _first be configured: https://pycloudlib.readthedocs.io/en/latest/configuration.html#configuration .. _Pytest marks: https://github.com/canonical/cloud-init/blob/af7eb1deab12c7208853c5d18b55228e0ba29c4d/tests/integration_tests/conftest.py#L220-L224 .. _IntegrationCloud: https://github.com/canonical/cloud-init/blob/af7eb1deab12c7208853c5d18b55228e0ba29c4d/tests/integration_tests/clouds.py#L102 +.. _pycloudlib configuration documentation: https://pycloudlib.readthedocs.io/en/latest/configuration.html From 4301c87d4605c17e5ad923f7d7f97c695568b998 Mon Sep 17 00:00:00 2001 From: a-dubs Date: Fri, 26 Jul 2024 14:23:10 -0400 Subject: [PATCH 2/4] initial review changes --- doc/rtd/development/integration_tests.rst | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/doc/rtd/development/integration_tests.rst b/doc/rtd/development/integration_tests.rst index 7b1055797da..f304339630a 100644 --- a/doc/rtd/development/integration_tests.rst +++ b/doc/rtd/development/integration_tests.rst @@ -62,7 +62,7 @@ All possible configuration values are defined in `tests/integration_tests/integration_settings.py`_. Look in this file for the full list of variables that are available and for context on what each variable does and what the default values are. -by supplying values in :file:`tests/integration_tests/user_settings.py` or by +Defaults can be overriden by supplying values in :file:`tests/integration_tests/user_settings.py` or by providing an environment variable of the same name prepended with ``CLOUD_INIT_``. For example, to set the ``PLATFORM`` setting: @@ -132,8 +132,6 @@ By default, the logs are collected to the ``/tmp/cloud_init_test_logs`` directory. To change the directory, set the ``LOCAL_LOG_PATH`` variable to the desired path. -TODO: can the collect log path be set to a relative path? - .. tab-set:: .. tab-item:: Inline environment variable @@ -177,18 +175,6 @@ Addtionally, for profiling the integration tests, set the ``INCLUDE_PROFILE`` variable to ``True``. This will generate a profile report for the integration test run, and the report will be stored in the directory specified by ``LOCAL_LOG_PATH``. -# We default our coverage to False because it involves modifying the -# cloud-init systemd services, which is too intrusive of a change to -# enable by default. If changed to true, the test directory corresponding -# to the test run under LOCAL_LOG_PATH defined above will contain an -# `html` directory with the coverage report. -INCLUDE_COVERAGE = False - -# We default our profile to False because it involves modifying the -# cloud-init systemd services, which is too intrusive of a change to -# enable by default. If changed to true, the test directory corresponding -# to the test run under LOCAL_LOG_PATH defined above will contain a report -INCLUDE_PROFILE = False .. tab-set:: @@ -273,10 +259,10 @@ tests against the image in question. If it's a RHEL8 image, then we would expect Ubuntu-specific tests to fail (and vice versa). To address this, a full image specification can be given. This is of -the form: ``[::[::]]`` where ``image_id`` is a +the form: ``[::::::]`` where ``image_id`` is a cloud's image ID, ``os`` is the OS name, and ``release`` is the OS release name. So, for example, Ubuntu 18.04 (Bionic Beaver) on LXD is -``ubuntu:bionic::ubuntu::bionic`` or RHEL8 on Amazon is +``ubuntu:jammy::ubuntu::jammy::22.04`` or RHEL8 on Amazon is ``ami-justanexample::rhel::8``. When a full specification is given, only tests which are intended for use on that OS and release will be executed. @@ -364,7 +350,7 @@ The ``client`` fixture should be used for most test cases. It ensures: ``module_client`` and ``class_client`` fixtures also exist for the purpose of running multiple tests against a single launched instance. They provide the exact same functionality as ``client``, but are -scoped to the module or class respectively. +scoped to the module or class respectively.ci ``session_cloud`` ----------------- From ae877dec62bc2d5aa16b15506a3fc1c79fef02c3 Mon Sep 17 00:00:00 2001 From: a-dubs Date: Wed, 31 Jul 2024 10:08:37 -0400 Subject: [PATCH 3/4] still needs a bit of work but nearly good enough for now --- doc/rtd/development/integration_tests.rst | 51 +++++++++++++++-------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/doc/rtd/development/integration_tests.rst b/doc/rtd/development/integration_tests.rst index f304339630a..df7d87f9898 100644 --- a/doc/rtd/development/integration_tests.rst +++ b/doc/rtd/development/integration_tests.rst @@ -27,32 +27,30 @@ Test execution ============== Test execution happens via ``pytest``. A ``tox`` definition exists to run -integration tests. To run all integration tests, you would run: +integration tests. When using this, normal ``pytest`` arguments can be +passed to the ``tox`` command by appending them after the ``--``. See the +following commands for examples. -.. code-block:: bash - - tox -e integration-tests - -``pytest`` arguments may also be passed. +.. tab-set:: -Or, specify a path to run all tests inside the given file or directory: + .. tab-item:: All integration tests -.. code-block:: bash + .. code-block:: bash - tox -e integration-tests tests/integration_tests/modules/test_combined.py + tox -e integration-tests + .. tab-item:: Tests inside file or directory -.. code-block:: bash + .. code-block:: bash - tox -e integration-tests tests/integration_tests/modules + tox -e integration-tests tests/integration_tests/modules/test_combined.py -Or, run a specific test: + .. tab-item:: A specific test -.. code-block:: bash + .. code-block:: bash - tox -e integration-tests tests/integration_tests/modules/test_combined.py::test_bootcmd + tox -e integration-tests tests/integration_tests/modules/test_combined.py::test_bootcmd -TODO: Maybe make these 3 commands in one tab group?? Configuration @@ -62,7 +60,8 @@ All possible configuration values are defined in `tests/integration_tests/integration_settings.py`_. Look in this file for the full list of variables that are available and for context on what each variable does and what the default values are. -Defaults can be overriden by supplying values in :file:`tests/integration_tests/user_settings.py` or by +Defaults can be overriden by supplying values in +:file:`tests/integration_tests/user_settings.py` or by providing an environment variable of the same name prepended with ``CLOUD_INIT_``. For example, to set the ``PLATFORM`` setting: @@ -262,11 +261,29 @@ To address this, a full image specification can be given. This is of the form: ``[::::::]`` where ``image_id`` is a cloud's image ID, ``os`` is the OS name, and ``release`` is the OS release name. So, for example, Ubuntu 18.04 (Bionic Beaver) on LXD is -``ubuntu:jammy::ubuntu::jammy::22.04`` or RHEL8 on Amazon is +``ubuntu::jammy::ubuntu::jammy::22.04`` or RHEL8 on Amazon is ``ami-justanexample::rhel::8``. When a full specification is given, only tests which are intended for use on that OS and release will be executed. +To run integration tests on a specific image, modify the ``OS_IMAGE`` +variable to be the desired image specification. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_OS_IMAGE='jammy' tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + OS_IMAGE = 'jammy' + + Image setup =========== From a08b7b3cfa969f9b071a279ba6c4fb56a42a0880 Mon Sep 17 00:00:00 2001 From: a-dubs Date: Fri, 2 Aug 2024 13:40:05 -0400 Subject: [PATCH 4/4] we should be good to go now --- doc/rtd/development/integration_tests.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/rtd/development/integration_tests.rst b/doc/rtd/development/integration_tests.rst index df7d87f9898..aecb0224455 100644 --- a/doc/rtd/development/integration_tests.rst +++ b/doc/rtd/development/integration_tests.rst @@ -102,8 +102,9 @@ Use in-place cloud-init source code The simplest way to test an integraton test using your current cloud-init changes is to set the ``CLOUD_INIT_SOURCE`` to ``IN_PLACE``. This works ONLY on LXD containers. This will mount the source code as-is directly into -the container to override the pre-existing cloudinit module. This -won't work for non-local LXD remotes and won't run any installation code. +the container to override the pre-existing cloud-init code within the +container. This won't work for non-local LXD remotes and won't run any +installation code since the source code is mounted directly. .. tab-set:: @@ -260,8 +261,8 @@ would expect Ubuntu-specific tests to fail (and vice versa). To address this, a full image specification can be given. This is of the form: ``[::::::]`` where ``image_id`` is a cloud's image ID, ``os`` is the OS name, and ``release`` is the OS -release name. So, for example, Ubuntu 18.04 (Bionic Beaver) on LXD is -``ubuntu::jammy::ubuntu::jammy::22.04`` or RHEL8 on Amazon is +release name. So, for example, Ubuntu 24.04 LTS (Noble Numbat) on LXD is +``ubuntu:noble::ubuntu::noble::24.04`` or RHEL8 on Amazon is ``ami-justanexample::rhel::8``. When a full specification is given, only tests which are intended for use on that OS and release will be executed.