-
Notifications
You must be signed in to change notification settings - Fork 180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
QE: Improve Cobbler tests and add a new profile test #6628
Conversation
1891147
to
0c85371
Compare
afc65d2
to
69e7f92
Compare
With my latest commit 69e7f92 I see the following Python error: Scenario: Modify Cobbler profile # features/dom.feature:15
This scenario ran at: 2023-02-20 17:22:39 +0100
Given cobblerd is running # features/step_definitions/cobbler_steps.rb:7
And I set "parent_distro1" as value of attribute "Distribution" for profile "parent_profile2" as user "testing" with password "testing" # features/step_definitions/cobbler_steps.rb:232
Modifying profile failed. <class 'AttributeError'>:'dict' object has no attribute 'startswith' (StandardError)
./features/support/cobbler_test.rb:258:in `rescue in profile_modify'
./features/support/cobbler_test.rb:255:in `profile_modify'
./features/step_definitions/cobbler_steps.rb:237:in `/^I set "([^"]*)" as value of attribute "([^"]*)" for profile "([^"]*)" as user "([^"]*)" with password "([^"]*)"$/'
features/dom.feature:17:in `I set "parent_distro1" as value of attribute "Distribution" for profile "parent_profile2" as user "testing" with password "testing"' I implemented the new |
@nodeg This is a Python error message. You should be able to see the same error message in the logfile. From there on you can just give me a call and we can have a look together. |
Here is the output from the log: [ThreadPoolExecutor-3_1] 2023-02-21T09:46:02 - INFO | Push successful
[ThreadPoolExecutor-3_13] 2023-02-21T09:46:02 - INFO | Push successful
[2023-02-21_094600_sync] 2023-02-21T09:46:02 - INFO | ### TASK COMPLETE ###
[Thread-137] 2023-02-21T09:48:24 - INFO | authenticate; ['testing', True]
[Thread-139] 2023-02-21T09:48:24 - INFO | REMOTE get_item(profile,parent_profile2); user(?)
[Thread-140] 2023-02-21T09:48:24 - INFO | Exception occurred: <class 'AttributeError'>
[Thread-140] 2023-02-21T09:48:24 - INFO | Exception value: 'dict' object has no attribute 'startswith'
[Thread-140] 2023-02-21T09:48:24 - INFO | Exception Info:
File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 3753, in _dispatch
return method_handle(*params)
File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 1874, in modify_profile
return self.modify_item("profile", object_id, attribute, arg, token)
File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 1830, in modify_item
obj = self.__get_object(object_id)
File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 706, in __get_object
if object_id.startswith("___NEW___"): |
f74a984
to
b35d4c0
Compare
a284949
to
b98c905
Compare
My fix for the TODO above worked: 14:35:15 @scope_cobbler
14:35:15 Feature: Cobbler editing profiles results in ISE
14:35:15 Tests for occuring Cobbler issue (bsc#1207532)
14:35:15 This feature is split up into 2 section
14:35:15 The first section uses the webUI
14:35:15 The second one uses the XML-RPC API
14:35:15
14:35:15 Background: The Cobbler service should be running # features/secondary/srv_cobbler_profile_ise.feature:11
14:35:15 This scenario ran at: 2023-03-09 14:34:50 +0100
14:35:15 Given cobblerd is running # features/step_definitions/cobbler_steps.rb:7
14:35:15
14:35:15 Scenario: Copy cobbler profiles on the server # features/secondary/srv_cobbler_profile_ise.feature:14
14:35:15 When I copy autoinstall mocked files on server # features/step_definitions/cobbler_steps.rb:231
14:35:15 This scenario took: 3 seconds
14:35:15
14:35:15 Scenario: Log in as testing user # features/secondary/srv_cobbler_profile_ise.feature:17
14:35:15 This scenario ran at: 2023-03-09 14:34:53 +0100
14:35:15 Given I am authorized as "testing" with password "testing" # features/step_definitions/navigation_steps.rb:515
14:35:15 This scenario took: 4 seconds
14:35:15
14:35:15 Scenario: Create a Cobbler distribution via the UI # features/secondary/srv_cobbler_profile_ise.feature:21
14:35:15 This scenario ran at: 2023-03-09 14:34:57 +0100
14:35:15 When I follow the left menu "Systems > Autoinstallation > Distributions" # features/step_definitions/navigation_steps.rb:348
14:35:15 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:35:15 And I follow "Create Distribution" # features/step_definitions/navigation_steps.rb:295
14:35:15 When I enter "isedistro_ui" as "label" # features/step_definitions/navigation_steps.rb:237
14:35:15 And I enter "/var/autoinstall/Fedora_12_i386/" as "basepath" # features/step_definitions/navigation_steps.rb:237
14:35:15 And I select "Fedora" from "installtype" # features/step_definitions/navigation_steps.rb:182
14:35:15 And I click on "Create Autoinstallable Distribution" # features/step_definitions/navigation_steps.rb:267
14:35:15 Then I should see a "Autoinstallable Distributions" text # features/step_definitions/navigation_steps.rb:583
14:35:15 And I should see a "isedistro_ui" link # features/step_definitions/navigation_steps.rb:638
14:35:15 This scenario took: 7 seconds
14:35:15
14:35:15 Scenario: Create a Cobbler profile via the UI # features/secondary/srv_cobbler_profile_ise.feature:31
14:35:15 This scenario ran at: 2023-03-09 14:35:04 +0100
14:35:15 When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:35:15 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:35:15 And I follow "Create Kickstart Profile" # features/step_definitions/navigation_steps.rb:295
14:35:15 When I enter "iseprofile_ui" as "kickstartLabel" # features/step_definitions/navigation_steps.rb:237
14:35:15 And I click on "Next" # features/step_definitions/navigation_steps.rb:267
14:35:15 And I click on "Next" # features/step_definitions/navigation_steps.rb:267
14:35:15 And I enter "linux" as "rootPassword" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I enter "linux" as "rootPasswordConfirm" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I click on "Finish" # features/step_definitions/navigation_steps.rb:267
14:38:39 Then I should see a "Autoinstallation: iseprofile_ui" text # features/step_definitions/navigation_steps.rb:583
14:38:39 And I should see a "Autoinstallation Details" link # features/step_definitions/navigation_steps.rb:638
14:38:39 This scenario took: 23 seconds
14:38:39
14:38:39 Scenario: Check Cobbler created distro and profile via the UI # features/secondary/srv_cobbler_profile_ise.feature:43
14:38:39 This scenario ran at: 2023-03-09 14:35:27 +0100
14:38:39 When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39 Then I should see a "iseprofile_ui" text # features/step_definitions/navigation_steps.rb:583
14:38:39 And I should see a "isedistro_ui" text # features/step_definitions/navigation_steps.rb:583
14:38:39 This scenario took: 3 seconds
14:38:39
14:38:39 Scenario: Change profile variables using the UI # features/secondary/srv_cobbler_profile_ise.feature:48
14:38:39 This scenario ran at: 2023-03-09 14:35:30 +0100
14:38:39 When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39 Timeout: Waiting AJAX transition (click link)
14:38:39 And I follow "iseprofile_ui" # features/step_definitions/navigation_steps.rb:295
14:38:39 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39 And I follow "Variables" # features/step_definitions/navigation_steps.rb:295
14:38:39 And I enter "ise_ui_test=ISE_UI_TEST" as "variables" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I click on "Update Variables" # features/step_definitions/navigation_steps.rb:267
14:38:39 And I refresh the page # features/step_definitions/navigation_steps.rb:1058
14:38:39 Then I should see a "ISE_UI_TEST" text # features/step_definitions/navigation_steps.rb:583
14:38:39 This scenario took: 35 seconds
14:38:39
14:38:39 Scenario: Create a Cobbler distribution via the UI in the XML-RPC context # features/secondary/srv_cobbler_profile_ise.feature:58
14:38:39 This scenario ran at: 2023-03-09 14:36:05 +0100
14:38:39 When I follow the left menu "Systems > Autoinstallation > Distributions" # features/step_definitions/navigation_steps.rb:348
14:38:39 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39 And I follow "Create Distribution" # features/step_definitions/navigation_steps.rb:295
14:38:39 When I enter "isedistro_api" as "label" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I enter "/var/autoinstall/Fedora_12_i386/" as "basepath" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I select "Fedora" from "installtype" # features/step_definitions/navigation_steps.rb:182
14:38:39 And I click on "Create Autoinstallable Distribution" # features/step_definitions/navigation_steps.rb:267
14:38:39 Then I should see a "Autoinstallable Distributions" text # features/step_definitions/navigation_steps.rb:583
14:38:39 And I should see a "isedistro_api" link # features/step_definitions/navigation_steps.rb:638
14:38:39 This scenario took: 8 seconds
14:38:39
14:38:39 Scenario: Create a Cobbler profile via the UI in the XML-RPC context # features/secondary/srv_cobbler_profile_ise.feature:68
14:38:39 This scenario ran at: 2023-03-09 14:36:13 +0100
14:38:39 When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39 And I follow "Create Kickstart Profile" # features/step_definitions/navigation_steps.rb:295
14:38:39 When I enter "iseprofile_api" as "kickstartLabel" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I click on "Next" # features/step_definitions/navigation_steps.rb:267
14:38:39 And I click on "Next" # features/step_definitions/navigation_steps.rb:267
14:38:39 And I enter "linux" as "rootPassword" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I enter "linux" as "rootPasswordConfirm" # features/step_definitions/navigation_steps.rb:237
14:38:39 And I click on "Finish" # features/step_definitions/navigation_steps.rb:267
14:38:39 Then I should see a "Autoinstallation: iseprofile_api" text # features/step_definitions/navigation_steps.rb:583
14:38:39 And I should see a "Autoinstallation Details" link # features/step_definitions/navigation_steps.rb:638
14:38:39 This scenario took: 23 seconds
14:38:39
14:38:39 Scenario: Check Cobbler created distro and profile via the UI in the XML-RPC context # features/secondary/srv_cobbler_profile_ise.feature:80
14:38:39 This scenario ran at: 2023-03-09 14:36:36 +0100
14:38:39 When I follow the left menu "Systems > Autoinstallation > Profiles" # features/step_definitions/navigation_steps.rb:348
14:38:39 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:38:39 Then I should see a "iseprofile_api" text # features/step_definitions/navigation_steps.rb:583
14:38:39 And I should see a "isedistro_api" text # features/step_definitions/navigation_steps.rb:583
14:38:39 This scenario took: 3 seconds
14:38:39
14:38:39 Scenario: Create a Cobbler system via the XML-RPC API # features/secondary/srv_cobbler_profile_ise.feature:85
14:38:39 This scenario ran at: 2023-03-09 14:36:39 +0100
14:38:39 And I create a system record with name "isesystem_api" and kickstart label "iseprofile_api" # features/step_definitions/api_common.rb:70
14:38:39 This scenario took: 117 seconds
14:38:39
14:38:39 Scenario: Create and modify a System profile using the XML-RPC API # features/secondary/srv_cobbler_profile_ise.feature:88
14:38:39 This scenario ran at: 2023-03-09 14:38:36 +0100
14:38:39 system_id: 1000010008
14:38:51 #<Thread:0x000055784fb64578@/root/spacewalk/testsuite/features/support/api_test.rb:64 run> terminated with exception (report_on_exception is true):
14:38:51 /root/spacewalk/testsuite/features/support/http_client.rb:99:in `call': API failure: No such cobbler system record (RuntimeError)
14:38:51 from /root/spacewalk/testsuite/features/support/api_test.rb:68:in `block (2 levels) in call'
14:38:51 from /root/spacewalk/testsuite/features/support/api_test.rb:65:in `synchronize'
14:38:51 from /root/spacewalk/testsuite/features/support/api_test.rb:65:in `block in call'
14:38:51 # XML-RPC should return an error here
14:38:51 And I create and modify the kickstart system "isesystem_api" with hostname "ise-system.test" via XML-RPC # features/step_definitions/api_common.rb:579
14:38:51 | inst.repo | http://ise.cobbler.test/ |
14:38:51 | self_update | http://ise.cobbler.test/ |
14:38:51 API failure: No such cobbler system record (RuntimeError)
14:38:51 ./features/support/http_client.rb:99:in `call'
14:38:51 ./features/support/api_test.rb:68:in `block (2 levels) in call'
14:38:51 ./features/support/api_test.rb:65:in `synchronize'
14:38:51 ./features/support/api_test.rb:65:in `block in call'
14:38:51 features/secondary/srv_cobbler_profile_ise.feature:90:in `I create and modify the kickstart system "isesystem_api" with hostname "ise-system.test" via XML-RPC'
(...)
14:39:27 This scenario took: 2 seconds
14:39:27
14:39:27 Scenario: Cleanup: delete test distros and profiles # features/secondary/srv_cobbler_profile_ise.feature:94
14:39:27 This scenario ran at: 2023-03-09 14:38:54 +0100
14:39:27 When I remove kickstart profiles and distros # features/step_definitions/cobbler_steps.rb:69
14:39:27 And I follow the left menu "Systems > System List" # features/step_definitions/navigation_steps.rb:348
14:39:27 WARN: Step ends with an ajax transition not finished, let's wait a bit!
14:39:27 And I wait until I see the "isesystem_api" system, refreshing the page # features/step_definitions/navigation_steps.rb:117
14:39:27 And I follow "isesystem_api" # features/step_definitions/navigation_steps.rb:295
14:39:27 And I follow "Delete System" # features/step_definitions/navigation_steps.rb:295
14:39:27 Then I should see a "Confirm System Profile Deletion" text # features/step_definitions/navigation_steps.rb:583
14:39:27 When I click on "Delete Profile" # features/step_definitions/navigation_steps.rb:267
14:39:27 And I wait until I see "has been deleted" text # features/step_definitions/navigation_steps.rb:39
14:39:27 This scenario took: 33 seconds
14:39:27
14:39:27 Failing Scenarios:
14:39:27 cucumber features/secondary/srv_cobbler_profile_ise.feature:88 # Scenario: Create and modify a System profile using the XML-RPC API |
The PR is finished now, I got a good PR test run except for the PXE boot minion, but we see those PXE issues in the normal CI test suite as well very often. |
I renamed the Cobbler distro test file to be more consistent with the other names of the feature files. Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
This will add specific Cobbler profile tests and extends our test suite to be able to use more Cobbler and SUMA XML-RPC API calls. See SUSE/spacewalk#20409 Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
This could interfere with other Cobbler tests. If we do not have proper clenaups a scenario can fail because of a distro with the same name. Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
We cannot successfully reproduce Cobbler race conditions so we do not want them to occur unwillingly. Therefore I moved all Cobbler tests from `secondary_parallelizable.yml` to to `secondary.yml`. Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
This will add the missing `@sle_minion` tag to the power management tests that require a SLE minion. Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
Specify that the XML-API test here will fail until Enno provides a fix for it. Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks for the test!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything looks good to me, thanks for that PR!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super tiny nitpick but optional. Great job with this PR 💯
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
very tiny nitpicks but looks good to me overall, great job with this 😁
Signed-off-by: Dominik Gedon <dominik.gedon@suse.com>
What does this PR change?
Improvements
This will
testsuite/features/secondary/srv_distro_cobbler.feature
totestsuite/features/secondary/srv_cobbler_distro.feature
to be more consistentI synchronize the tftp configuration on the proxy with the server
toI run Cobbler sync (with|without) error checking
since it only ran acobbler
sync and checks for errorssrv_cobbler_sync.feature
sle_micro
tag to thesrv_power_management_XXX.feature
files because they need a SLE minion to worksecondary_parallelizable.yml
to tosecondary.yml
due to the fact that we cannot reproduce race conditions (e.g. with Cobbler sync) willingly and consistentlytestdistro
withbuildisodistro
in the buildiso tests to be more idempotent (to prevent):New tests
The additions to the
CobblerTest
class are just added to the very end so that @elariekerboull has no major problems rebasing his PR for refactoring theCobblerTest
class.srv_cobbler_profile_ise.feature
will fail (what is the goal right now) until @SchoolGuy fixes the issue behind it with #6676TODO
GUI diff
No difference.
Documentation
No documentation needed: only internal and user invisible changes
DONE
Test coverage
Cucumber tests were added
DONE
Links
get_*_handle()
methods are missing required parameter cobbler/cobbler#3356Changelogs
Make sure the changelogs entries you are adding are compliant with https://github.com/uyuni-project/uyuni/wiki/Contributing#changelogs and https://github.com/uyuni-project/uyuni/wiki/Contributing#uyuni-projectuyuni-repository
If you don't need a changelog check, please mark this checkbox:
If you uncheck the checkbox after the PR is created, you will need to re-run
changelog_test
(see below)Re-run a test
If you need to re-run a test, please mark the related checkbox, it will be unchecked automatically once it has re-run: