Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QE: Improve Cobbler tests and add a new profile test #6628

Merged
merged 9 commits into from
Mar 13, 2023

Conversation

nodeg
Copy link
Member

@nodeg nodeg commented Feb 17, 2023

What does this PR change?

Improvements

This will

  • rename testsuite/features/secondary/srv_distro_cobbler.feature to testsuite/features/secondary/srv_cobbler_distro.feature to be more consistent
  • rename and refactor the step I synchronize the tftp configuration on the proxy with the server to I run Cobbler sync (with|without) error checking since it only ran a cobbler sync and checks for errors
  • add a missing Cobbler tag to srv_cobbler_sync.feature
  • add missing sle_micro tag to the srv_power_management_XXX.feature files because they need a SLE minion to work
  • moved all Cobbler tests from secondary_parallelizable.yml to to secondary.yml due to the fact that we cannot reproduce race conditions (e.g. with Cobbler sync) willingly and consistently
  • rename testdistro with buildisodistro in the buildiso tests to be more idempotent (to prevent):
    image

New tests

  • extends our test suite with more calls to the Cobbler and the SUMA XML-RPC API (with regards to Cobbler)
  • add Cobbler profile tests, specifically for testing 1 common L3s. It will create profiles and modify profiles via the web UI and the XML-RPC API

The additions to the CobblerTest class are just added to the very end so that @elariekerboull has no major problems rebasing his PR for refactoring the CobblerTest class.

⚠️ Important: The new feature srv_cobbler_profile_ise.feature will fail (what is the goal right now) until @SchoolGuy fixes the issue behind it with #6676

TODO

  • check why the normal Cobbler tests fail
  • check why the cleanup scenario fails. There is a small delay when going to Systems -> Systems List

GUI diff

No difference.

  • DONE

Documentation

  • No documentation needed: only internal and user invisible changes

  • DONE

Test coverage

  • Cucumber tests were added

  • DONE

Links

Changelogs

Make sure the changelogs entries you are adding are compliant with https://github.com/uyuni-project/uyuni/wiki/Contributing#changelogs and https://github.com/uyuni-project/uyuni/wiki/Contributing#uyuni-projectuyuni-repository

If you don't need a changelog check, please mark this checkbox:

  • No changelog needed

If you uncheck the checkbox after the PR is created, you will need to re-run changelog_test (see below)

Re-run a test

If you need to re-run a test, please mark the related checkbox, it will be unchecked automatically once it has re-run:

  • Re-run test "changelog_test"
  • Re-run test "backend_unittests_pgsql"
  • Re-run test "java_pgsql_tests"
  • Re-run test "schema_migration_test_pgsql"
  • Re-run test "susemanager_unittests"
  • Re-run test "javascript_lint"
  • Re-run test "spacecmd_unittests"

@nodeg nodeg force-pushed the qe-cobbler-l3 branch 2 times, most recently from afc65d2 to 69e7f92 Compare February 20, 2023 16:25
@nodeg
Copy link
Member Author

nodeg commented Feb 20, 2023

With my latest commit 69e7f92 I see the following Python error:

  Scenario: Modify Cobbler profile                                                                                                          # features/dom.feature:15                                                                                                                                                           
      This scenario ran at: 2023-02-20 17:22:39 +0100                                                                                                           
    Given cobblerd is running                                                                                                               # features/step_definitions/cobbler_steps.rb:7                                                                                                                                      
    And I set "parent_distro1" as value of attribute "Distribution" for profile "parent_profile2" as user "testing" with password "testing" # features/step_definitions/cobbler_steps.rb:232                                                                                                                                    
      Modifying profile failed. <class 'AttributeError'>:'dict' object has no attribute 'startswith' (StandardError)                 
      ./features/support/cobbler_test.rb:258:in `rescue in profile_modify'                                                                                      
      ./features/support/cobbler_test.rb:255:in `profile_modify'                                                                                                
      ./features/step_definitions/cobbler_steps.rb:237:in `/^I set "([^"]*)" as value of attribute "([^"]*)" for profile "([^"]*)" as user "([^"]*)" with password "([^"]*)"$/'                                                                                                                                                 
      features/dom.feature:17:in `I set "parent_distro1" as value of attribute "Distribution" for profile "parent_profile2" as user "testing" with password "testing"' 

I implemented the new modify_profile method with regards to https://cobbler.readthedocs.io/en/latest/code-autodoc/cobbler.html#cobbler.remote.CobblerXMLRPCInterface.modify_profile

@nodeg nodeg added the cobbler label Feb 20, 2023
@SchoolGuy
Copy link
Contributor

@nodeg This is a Python error message. You should be able to see the same error message in the logfile. From there on you can just give me a call and we can have a look together.

@nodeg
Copy link
Member Author

nodeg commented Feb 21, 2023

Here is the output from the log:

[ThreadPoolExecutor-3_1] 2023-02-21T09:46:02 - INFO | Push successful
[ThreadPoolExecutor-3_13] 2023-02-21T09:46:02 - INFO | Push successful
[2023-02-21_094600_sync] 2023-02-21T09:46:02 - INFO | ### TASK COMPLETE ###
[Thread-137] 2023-02-21T09:48:24 - INFO | authenticate; ['testing', True]
[Thread-139] 2023-02-21T09:48:24 - INFO | REMOTE get_item(profile,parent_profile2); user(?)
[Thread-140] 2023-02-21T09:48:24 - INFO | Exception occurred: <class 'AttributeError'>
[Thread-140] 2023-02-21T09:48:24 - INFO | Exception value: 'dict' object has no attribute 'startswith'
[Thread-140] 2023-02-21T09:48:24 - INFO | Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 3753, in _dispatch
    return method_handle(*params)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 1874, in modify_profile
    return self.modify_item("profile", object_id, attribute, arg, token)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 1830, in modify_item
    obj = self.__get_object(object_id)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 706, in __get_object
    if object_id.startswith("___NEW___"):

@nodeg nodeg force-pushed the qe-cobbler-l3 branch 2 times, most recently from a284949 to b98c905 Compare February 23, 2023 13:40
@nodeg
Copy link
Member Author

nodeg commented Mar 9, 2023

My fix for the TODO above worked:

14:35:15  @scope_cobbler
14:35:15  Feature: Cobbler editing profiles results in ISE
14:35:15  Tests for occuring Cobbler issue (bsc#1207532)
14:35:15  This feature is split up into 2 section
14:35:15  The first section uses the webUI
14:35:15  The second one uses the XML-RPC API
14:35:15  
14:35:15  Background: The Cobbler service should be running # features/secondary/srv_cobbler_profile_ise.feature:11
14:35:15  This scenario ran at: 2023-03-09 14:34:50 +0100
14:35:15  Given cobblerd is running                       # features/step_definitions/cobbler_steps.rb:7
14:35:15  
14:35:15  Scenario: Copy cobbler profiles on the server    # features/secondary/srv_cobbler_profile_ise.feature:14
14:35:15  When I copy autoinstall mocked files on server # features/step_definitions/cobbler_steps.rb:231
14:35:15  This scenario took: 3 seconds
14:35:15  
14:35:15  Scenario: Log in as testing user                             # features/secondary/srv_cobbler_profile_ise.feature:17
14:35:15  This scenario ran at: 2023-03-09 14:34:53 +0100
14:35:15  Given I am authorized as "testing" with password "testing" # features/step_definitions/navigation_steps.rb:515
14:35:15  This scenario took: 4 seconds
14:35:15  
14:35:15  Scenario: Create a Cobbler distribution via the UI                         # features/secondary/srv_cobbler_profile_ise.feature:21
14:35:15  This scenario ran at: 2023-03-09 14:34:57 +0100
14:35:15  When I follow the left menu "Systems > Autoinstallation > Distributions" # features/step_definitions/navigation_steps.rb:348
14:35:15  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:35:15  And I follow "Create Distribution"                                       # features/step_definitions/navigation_steps.rb:295
14:35:15  When I enter "isedistro_ui" as "label"                                   # features/step_definitions/navigation_steps.rb:237
14:35:15  And I enter "/var/autoinstall/Fedora_12_i386/" as "basepath"             # features/step_definitions/navigation_steps.rb:237
14:35:15  And I select "Fedora" from "installtype"                                 # features/step_definitions/navigation_steps.rb:182
14:35:15  And I click on "Create Autoinstallable Distribution"                     # features/step_definitions/navigation_steps.rb:267
14:35:15  Then I should see a "Autoinstallable Distributions" text                 # features/step_definitions/navigation_steps.rb:583
14:35:15  And I should see a "isedistro_ui" link                                   # features/step_definitions/navigation_steps.rb:638
14:35:15  This scenario took: 7 seconds
14:35:15  
14:35:15  Scenario: Create a Cobbler profile via the UI                         # features/secondary/srv_cobbler_profile_ise.feature:31
14:35:15  This scenario ran at: 2023-03-09 14:35:04 +0100
14:35:15  When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:35:15  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:35:15  And I follow "Create Kickstart Profile"                             # features/step_definitions/navigation_steps.rb:295
14:35:15  When I enter "iseprofile_ui" as "kickstartLabel"                    # features/step_definitions/navigation_steps.rb:237
14:35:15  And I click on "Next"                                               # features/step_definitions/navigation_steps.rb:267
14:35:15  And I click on "Next"                                               # features/step_definitions/navigation_steps.rb:267
14:35:15  And I enter "linux" as "rootPassword"                               # features/step_definitions/navigation_steps.rb:237
14:38:39  And I enter "linux" as "rootPasswordConfirm"                        # features/step_definitions/navigation_steps.rb:237
14:38:39  And I click on "Finish"                                             # features/step_definitions/navigation_steps.rb:267
14:38:39  Then I should see a "Autoinstallation: iseprofile_ui" text          # features/step_definitions/navigation_steps.rb:583
14:38:39  And I should see a "Autoinstallation Details" link                  # features/step_definitions/navigation_steps.rb:638
14:38:39  This scenario took: 23 seconds
14:38:39  
14:38:39  Scenario: Check Cobbler created distro and profile via the UI         # features/secondary/srv_cobbler_profile_ise.feature:43
14:38:39  This scenario ran at: 2023-03-09 14:35:27 +0100
14:38:39  When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39  Then I should see a "iseprofile_ui" text                            # features/step_definitions/navigation_steps.rb:583
14:38:39  And I should see a "isedistro_ui" text                              # features/step_definitions/navigation_steps.rb:583
14:38:39  This scenario took: 3 seconds
14:38:39  
14:38:39  Scenario: Change profile variables using the UI                       # features/secondary/srv_cobbler_profile_ise.feature:48
14:38:39  This scenario ran at: 2023-03-09 14:35:30 +0100
14:38:39  When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39  Timeout: Waiting AJAX transition (click link)
14:38:39  And I follow "iseprofile_ui"                                        # features/step_definitions/navigation_steps.rb:295
14:38:39  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39  And I follow "Variables"                                            # features/step_definitions/navigation_steps.rb:295
14:38:39  And I enter "ise_ui_test=ISE_UI_TEST" as "variables"                # features/step_definitions/navigation_steps.rb:237
14:38:39  And I click on "Update Variables"                                   # features/step_definitions/navigation_steps.rb:267
14:38:39  And I refresh the page                                              # features/step_definitions/navigation_steps.rb:1058
14:38:39  Then I should see a "ISE_UI_TEST" text                              # features/step_definitions/navigation_steps.rb:583
14:38:39  This scenario took: 35 seconds
14:38:39  
14:38:39  Scenario: Create a Cobbler distribution via the UI in the XML-RPC context  # features/secondary/srv_cobbler_profile_ise.feature:58
14:38:39  This scenario ran at: 2023-03-09 14:36:05 +0100
14:38:39  When I follow the left menu "Systems > Autoinstallation > Distributions" # features/step_definitions/navigation_steps.rb:348
14:38:39  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39  And I follow "Create Distribution"                                       # features/step_definitions/navigation_steps.rb:295
14:38:39  When I enter "isedistro_api" as "label"                                  # features/step_definitions/navigation_steps.rb:237
14:38:39  And I enter "/var/autoinstall/Fedora_12_i386/" as "basepath"             # features/step_definitions/navigation_steps.rb:237
14:38:39  And I select "Fedora" from "installtype"                                 # features/step_definitions/navigation_steps.rb:182
14:38:39  And I click on "Create Autoinstallable Distribution"                     # features/step_definitions/navigation_steps.rb:267
14:38:39  Then I should see a "Autoinstallable Distributions" text                 # features/step_definitions/navigation_steps.rb:583
14:38:39  And I should see a "isedistro_api" link                                  # features/step_definitions/navigation_steps.rb:638
14:38:39  This scenario took: 8 seconds
14:38:39  
14:38:39  Scenario: Create a Cobbler profile via the UI in the XML-RPC context  # features/secondary/srv_cobbler_profile_ise.feature:68
14:38:39  This scenario ran at: 2023-03-09 14:36:13 +0100
14:38:39  When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39  And I follow "Create Kickstart Profile"                             # features/step_definitions/navigation_steps.rb:295
14:38:39  When I enter "iseprofile_api" as "kickstartLabel"                   # features/step_definitions/navigation_steps.rb:237
14:38:39  And I click on "Next"                                               # features/step_definitions/navigation_steps.rb:267
14:38:39  And I click on "Next"                                               # features/step_definitions/navigation_steps.rb:267
14:38:39  And I enter "linux" as "rootPassword"                               # features/step_definitions/navigation_steps.rb:237
14:38:39  And I enter "linux" as "rootPasswordConfirm"                        # features/step_definitions/navigation_steps.rb:237
14:38:39  And I click on "Finish"                                             # features/step_definitions/navigation_steps.rb:267
14:38:39  Then I should see a "Autoinstallation: iseprofile_api" text         # features/step_definitions/navigation_steps.rb:583
14:38:39  And I should see a "Autoinstallation Details" link                  # features/step_definitions/navigation_steps.rb:638
14:38:39  This scenario took: 23 seconds
14:38:39  
14:38:39  Scenario: Check Cobbler created distro and profile via the UI in the XML-RPC context # features/secondary/srv_cobbler_profile_ise.feature:80
14:38:39  This scenario ran at: 2023-03-09 14:36:36 +0100
14:38:39  When I follow the left menu "Systems > Autoinstallation > Profiles"                # features/step_definitions/navigation_steps.rb:348
14:38:39  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39  Then I should see a "iseprofile_api" text                                          # features/step_definitions/navigation_steps.rb:583
14:38:39  And I should see a "isedistro_api" text                                            # features/step_definitions/navigation_steps.rb:583
14:38:39  This scenario took: 3 seconds
14:38:39  
14:38:39  Scenario: Create a Cobbler system via the XML-RPC API                                         # features/secondary/srv_cobbler_profile_ise.feature:85
14:38:39  This scenario ran at: 2023-03-09 14:36:39 +0100
14:38:39  And I create a system record with name "isesystem_api" and kickstart label "iseprofile_api" # features/step_definitions/api_common.rb:70
14:38:39  This scenario took: 117 seconds
14:38:39  
14:38:39  Scenario: Create and modify a System profile using the XML-RPC API                                         # features/secondary/srv_cobbler_profile_ise.feature:88
14:38:39  This scenario ran at: 2023-03-09 14:38:36 +0100
14:38:39  system_id: 1000010008
14:38:51  #<Thread:0x000055784fb64578@/root/spacewalk/testsuite/features/support/api_test.rb:64 run> terminated with exception (report_on_exception is true):
14:38:51  /root/spacewalk/testsuite/features/support/http_client.rb:99:in `call': API failure: No such cobbler system record (RuntimeError)
14:38:51  from /root/spacewalk/testsuite/features/support/api_test.rb:68:in `block (2 levels) in call'
14:38:51  from /root/spacewalk/testsuite/features/support/api_test.rb:65:in `synchronize'
14:38:51  from /root/spacewalk/testsuite/features/support/api_test.rb:65:in `block in call'
14:38:51  # XML-RPC should return an error here
14:38:51  And I create and modify the kickstart system "isesystem_api" with hostname "ise-system.test" via XML-RPC # features/step_definitions/api_common.rb:579
14:38:51  | inst.repo   | http://ise.cobbler.test/ |
14:38:51  | self_update | http://ise.cobbler.test/ |
14:38:51        API failure: No such cobbler system record (RuntimeError)
14:38:51        ./features/support/http_client.rb:99:in `call'
14:38:51        ./features/support/api_test.rb:68:in `block (2 levels) in call'
14:38:51        ./features/support/api_test.rb:65:in `synchronize'
14:38:51        ./features/support/api_test.rb:65:in `block in call'
14:38:51        features/secondary/srv_cobbler_profile_ise.feature:90:in `I create and modify the kickstart system "isesystem_api" with hostname "ise-system.test" via XML-RPC'
(...)
14:39:27  This scenario took: 2 seconds
14:39:27  
14:39:27  Scenario: Cleanup: delete test distros and profiles                      # features/secondary/srv_cobbler_profile_ise.feature:94
14:39:27  This scenario ran at: 2023-03-09 14:38:54 +0100
14:39:27  When I remove kickstart profiles and distros                           # features/step_definitions/cobbler_steps.rb:69
14:39:27  And I follow the left menu "Systems > System List"                     # features/step_definitions/navigation_steps.rb:348
14:39:27  WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:39:27  And I wait until I see the "isesystem_api" system, refreshing the page # features/step_definitions/navigation_steps.rb:117
14:39:27  And I follow "isesystem_api"                                           # features/step_definitions/navigation_steps.rb:295
14:39:27  And I follow "Delete System"                                           # features/step_definitions/navigation_steps.rb:295
14:39:27  Then I should see a "Confirm System Profile Deletion" text             # features/step_definitions/navigation_steps.rb:583
14:39:27  When I click on "Delete Profile"                                       # features/step_definitions/navigation_steps.rb:267
14:39:27  And I wait until I see "has been deleted" text                         # features/step_definitions/navigation_steps.rb:39
14:39:27  This scenario took: 33 seconds
14:39:27  
14:39:27  Failing Scenarios:
14:39:27  cucumber features/secondary/srv_cobbler_profile_ise.feature:88 # Scenario: Create and modify a System profile using the XML-RPC API

@nodeg nodeg changed the title QE: Add Cobbler profile tests QE: Improve Cobbler tests and add a new profile test Mar 10, 2023
@nodeg nodeg marked this pull request as draft March 10, 2023 14:25
@nodeg
Copy link
Member Author

nodeg commented Mar 13, 2023

The PR is finished now, I got a good PR test run except for the PXE boot minion, but we see those PXE issues in the normal CI test suite as well very often.

@nodeg nodeg marked this pull request as ready for review March 13, 2023 08:10
I renamed the Cobbler distro test file to be more consistent with the
other names of the feature files.

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
This will add specific Cobbler profile tests and extends our test suite
to be able to use more Cobbler and SUMA XML-RPC API calls.

See SUSE/spacewalk#20409

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
This could interfere with other Cobbler tests. If we do not have proper
clenaups a scenario can fail because of a distro with the same name.

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
We cannot successfully reproduce Cobbler race conditions so we do not
want them to occur unwillingly. Therefore I moved all Cobbler tests from
`secondary_parallelizable.yml` to to `secondary.yml`.

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
This will add the missing `@sle_minion` tag to the power management
tests that require a SLE minion.

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Specify that the XML-API test here will fail until Enno provides a fix
for it.

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Copy link
Contributor

@SchoolGuy SchoolGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the test!

Copy link
Contributor

@elariekerboull elariekerboull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good to me, thanks for that PR!

Copy link
Contributor

@ktsamis ktsamis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super tiny nitpick but optional. Great job with this PR 💯

testsuite/features/support/namespaces/system.rb Outdated Show resolved Hide resolved
Copy link
Contributor

@vandabarata vandabarata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very tiny nitpicks but looks good to me overall, great job with this 😁

Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
@nodeg nodeg merged commit 0cf4460 into uyuni-project:master Mar 13, 2023
@nodeg nodeg deleted the qe-cobbler-l3 branch March 13, 2023 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants