diff --git a/doc/rtd/development/integration_tests.rst b/doc/rtd/development/integration_tests.rst index 5fe5845dd4b..7b1055797da 100644 --- a/doc/rtd/development/integration_tests.rst +++ b/doc/rtd/development/integration_tests.rst @@ -31,26 +31,179 @@ integration tests. To run all integration tests, you would run: .. code-block:: bash - $ tox -e integration-tests + tox -e integration-tests -``pytest`` arguments may also be passed. For example: +``pytest`` arguments may also be passed. + +Or, specify a path to run all tests inside the given file or directory: + +.. code-block:: bash + + tox -e integration-tests tests/integration_tests/modules/test_combined.py + + +.. code-block:: bash + + tox -e integration-tests tests/integration_tests/modules + +Or, run a specific test: .. code-block:: bash - $ tox -e integration-tests tests/integration_tests/modules/test_combined.py + tox -e integration-tests tests/integration_tests/modules/test_combined.py::test_bootcmd + +TODO: Maybe make these 3 commands in one tab group?? + Configuration ============= All possible configuration values are defined in -`tests/integration_tests/integration_settings.py`_. Defaults can be overridden +`tests/integration_tests/integration_settings.py`_. Look in this file for +the full list of variables that are available and for context on what each +variable does and what the default values are. by supplying values in :file:`tests/integration_tests/user_settings.py` or by providing an environment variable of the same name prepended with ``CLOUD_INIT_``. For example, to set the ``PLATFORM`` setting: .. code-block:: bash - CLOUD_INIT_PLATFORM='ec2' pytest tests/integration_tests/ + CLOUD_INIT_PLATFORM='ec2' tox -e integration_tests -- tests/integration_tests/ + + +Common integration test run configurations +========================================== + + +Keep instance after test run +------------------------------- + +By default, the test instance is torn down after the test run. To keep +the instance running after the test run, set the ``KEEP_INSTANCE`` variable +to ``True``. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_KEEP_INSTANCE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + KEEP_INSTANCE = True + + +Use in-place cloud-init source code +------------------------------------- + +The simplest way to test an integraton test using your current cloud-init +changes is to set the ``CLOUD_INIT_SOURCE`` to ``IN_PLACE``. This works ONLY +on LXD containers. This will mount the source code as-is directly into +the container to override the pre-existing cloudinit module. This +won't work for non-local LXD remotes and won't run any installation code. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_CLOUD_INIT_SOURCE=IN_PLACE tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + CLOUD_INIT_SOURCE = 'IN_PLACE' + + +Collecting logs after test run +------------------------------- + +By default, logs are collected only when a test fails, by running ``cloud-init +collect-logs`` on the instance. To collect logs after every test run, set the +``COLLECT_LOGS`` variable to ``ALWAYS``. + +By default, the logs are collected to the ``/tmp/cloud_init_test_logs`` +directory. To change the directory, set the ``LOCAL_LOG_PATH`` variable to +the desired path. + +TODO: can the collect log path be set to a relative path? + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_COLLECT_LOGS=ALWAYS CLOUD_INIT_LOCAL_LOG_PATH=/tmp/your-local-directory tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + COLLECT_LOGS = "ALWAYS" + LOCAL_LOG_PATH = "/tmp/logs" + + +Advanced test reporting and profiling +------------------------------------- + +For advanced test reporting, set the ``INCLUDE_COVERAGE`` variable to ``True``. +This will generate a coverage report for the integration test run, and the +report will be stored in an ``html`` directory inside the directory specified +by ``LOCAL_LOG_PATH``. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_INCLUDE_COVERAGE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + INCLUDE_COVERAGE = True + + +Addtionally, for profiling the integration tests, set the ``INCLUDE_PROFILE`` +variable to ``True``. This will generate a profile report for the integration +test run, and the report will be stored in the directory specified by +``LOCAL_LOG_PATH``. +# We default our coverage to False because it involves modifying the +# cloud-init systemd services, which is too intrusive of a change to +# enable by default. If changed to true, the test directory corresponding +# to the test run under LOCAL_LOG_PATH defined above will contain an +# `html` directory with the coverage report. +INCLUDE_COVERAGE = False + +# We default our profile to False because it involves modifying the +# cloud-init systemd services, which is too intrusive of a change to +# enable by default. If changed to true, the test directory corresponding +# to the test run under LOCAL_LOG_PATH defined above will contain a report +INCLUDE_PROFILE = False + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_INCLUDE_PROFILE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + INCLUDE_PROFILE = True + Cloud interaction ================= @@ -65,6 +218,39 @@ For a minimal setup using LXD, write the following to [lxd] + +For more information on configuring pycloudlib, see the +`pycloudlib configuration documentation`_. + +To specify a specific cloud to test against, first, ensure that your pycloudlib +configuration is set up correctly. Then, modify the ``PLATFORM`` variable to be +on of: + +- ``azure``: Microsoft Azure +- ``ec2``: Amazon EC2 +- ``gce``: Google Compute Engine +- ``ibm``: IBM Cloud +- ``lxd_container``: LXD container +- ``lxd_vm``: LXD VM +- ``oci``: Oracle Cloud Infrastructure +- ``openstack``: OpenStack +- ``qemu``: QEMU + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_PLATFORM='lxd_container' tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + PLATFORM = 'lxd_container' + + Image selection =============== @@ -108,6 +294,29 @@ via fixture. Image setup roughly follows these steps: * Take a snapshot of the instance to be used as a new image from which new instances can be launched. + +Keep image after test run +-------------------------- + +By default, the image created during the test run is torn down after +the test run. If further debugging is needed, you can keep the image snapshot +for further use by setting the ``KEEP_IMAGE`` variable to ``True``. + +.. tab-set:: + + .. tab-item:: Inline environment variable + + .. code-block:: bash + + CLOUD_INIT_KEEP_IMAGE=True tox -e integration_tests + + .. tab-item:: user_settings.py file + + .. code-block:: python + + KEEP_IMAGE = True + + Test setup ========== @@ -213,3 +422,4 @@ Customizing the launch arguments before launching an instance manually: .. _first be configured: https://pycloudlib.readthedocs.io/en/latest/configuration.html#configuration .. _Pytest marks: https://github.com/canonical/cloud-init/blob/af7eb1deab12c7208853c5d18b55228e0ba29c4d/tests/integration_tests/conftest.py#L220-L224 .. _IntegrationCloud: https://github.com/canonical/cloud-init/blob/af7eb1deab12c7208853c5d18b55228e0ba29c4d/tests/integration_tests/clouds.py#L102 +.. _pycloudlib configuration documentation: https://pycloudlib.readthedocs.io/en/latest/configuration.html