Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

QE: Fix and improve the server hostname rename test #7700

Merged
merged 2 commits into from
Oct 20, 2023
Merged

QE: Fix and improve the server hostname rename test #7700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a3c5f97
QE: Move Salt high state step to correct place
nodeg Oct 20, 2023
acbd866
QE: Fix and improve server hostname rename test
nodeg Oct 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions testsuite/.rubocop_todo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,7 @@ Style/AndOr:
- 'features/step_definitions/common_steps.rb'
- 'features/step_definitions/file_management_steps.rb'
- 'features/step_definitions/retail_steps.rb'
- 'features/step_definitions/salt_steps.rb'
- 'features/step_definitions/setup_steps.rb'

# Offense count: 1
Expand Down
90 changes: 90 additions & 0 deletions testsuite/features/secondary/srv_rename_hostname.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,51 @@ Feature: Reconfigure the server's hostname
When I change the server's short hostname from hosts and hostname files
And I run spacewalk-hostname-rename command on the server

@proxy
Scenario: Copy the new server keys and configure the proxy
When I copy server's keys to the proxy
And I configure the proxy
Then I should see "proxy" via spacecmd
When I restart the "salt-minion" service on "proxy"
Then service "salt-minion" is active on "proxy"
When I restart the "salt-broker" service on "proxy"
Then service "salt-broker" is active on "proxy"

@proxy
Scenario: Apply high state on the proxy to populate new server CA
When I apply highstate on "proxy"

@sle_minion
Scenario: Apply high state on the SUSE Minion to populate new server CA
When I apply highstate on "sle_minion"

@ssh_minion
Scenario: Apply high state on the SUSE SSH Minion to populate new server CA
When I apply highstate on "ssh_minion"

@rhlike_minion
Scenario: Apply high state on the Red Hat-like Minion to populate new server CA
When I apply highstate on "rhlike_minion"

@deblike_minion
Scenario: Apply high state on the Debian-like Minion to populate new server CA
When I apply highstate on "deblike_minion"

@buildhost
Scenario: Apply high state on the build host to populate new server CA
When I apply highstate on "build_host"

@virthost_kvm
Scenario: Apply high state on the virthost to populate new server CA
When I apply highstate on "kvm_server"

@pxeboot_minion
Scenario: Apply high state on the PXE boot minion to populate new server CA
When I apply highstate on "pxeboot_minion"
Comment on lines +38 to +64
Copy link
Member

@srbarrios srbarrios Oct 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use SSM to apply high state to all the registered systems at the same time.
That way if we add any other system in our CI test suite later, we don't need to go here and maintain this part.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I still have issues with high state for the proxy when doing it with Salt:

  @proxy
  Scenario: Apply high state on the proxy to populate new server CA # features/secondary/srv_rename_hostname.feature:67
      This scenario ran at: 2023-10-18 09:37:30 +0200
    Given the Salt master can reach "proxy"                         # features/step_definitions/salt_steps.rb:11
      It took 1 seconds to contact the minion
    And I apply highstate on "proxy"                                # features/step_definitions/command_steps.rb:241
      Timeout after 250 seconds (repeat_until_timeout), last result was: bash: dominik-pxy.mgr.suse.de: command not found
       (RuntimeError)
      ./features/support/commonlib.rb:117:in `block in repeat_until_timeout'
      ./features/support/commonlib.rb:100:in `repeat_until_timeout'
      ./features/support/lavanda.rb:193:in `run_until_ok'
      ./features/step_definitions/command_steps.rb:248:in `/^I apply highstate on "([^"]*)"$/'
      features/secondary/srv_rename_hostname.feature:69:in `I apply highstate on "proxy"'

which uses

When(/^I apply highstate on "([^"]*)"$/) do |host|
  system_name = get_system_name(host)
  if host.include? 'ssh_minion'
    cmd = 'mgr-salt-ssh'
  elsif host.include? 'minion' or host.include? 'build'
    cmd = 'salt'
  end
  get_target('server').run_until_ok("#{cmd} #{system_name} state.highstate")
end

And running salt dominik-pxy.mgr.suse.de state.highstatemanually from the server works.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After wondering about SSM, I think we do not yet have a step to select all systems from the systems list and add them to the SSM.

Copy link
Member

@srbarrios srbarrios Oct 20, 2023 •
edited by nodeg
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here a new proposal scenario that will apply high-state in all registered systems:

  Scenario: Apply high-state to all systems registered
    When I follow the left menu "System > System List > All"
    And I click on "Select All"
    And I follow the left menu "Systems > System Set Manager > Overview"
    And I click on "Apply"
    And I click on "Apply Highstate"
    And I click on the clear SSM button

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. I will do some test run with this code.

Copy link
Member Author

@nodeg nodeg Oct 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  Scenario: Apply high-state to all systems registered                   # features/secondary/srv_rename_hostname.feature:34
      This scenario ran at: 2023-10-20 16:35:03 +0200
    When I follow the left menu "System > System List > All"             # features/step_definitions/navigation_steps.rb:363
      WARN: Step ends with an ajax transition not finished, let's wait a bit!
    And I click on "Select All"                                          # features/step_definitions/navigation_steps.rb:282
    And I follow the left menu "Systems > System Set Manager"            # features/step_definitions/navigation_steps.rb:363
      WARN: Step ends with an ajax transition not finished, let's wait a bit!
    And I click on "Apply"                                               # features/step_definitions/navigation_steps.rb:282
      Unable to find button "Apply" that is not disabled (Capybara::ElementNotFound)
      ./features/support/commonlib.rb:146:in `click_button_and_wait'
      ./features/step_definitions/navigation_steps.rb:283:in `/^I click on "([^"]*)"$/'
      features/secondary/srv_rename_hostname.feature:38:in `I click on "Apply"'
    And I click on "Apply Highstate"                                     # features/step_definitions/navigation_steps.rb:282
    Then I should see a "Applying the highstate has been scheduled" text # features/step_definitions/navigation_steps.rb:599
    And I click on the clear SSM button                                  # features/step_definitions/navigation_steps.rb:799
=> /var/log/rhn/rhn_web_ui.log

image

It seems no systems were selected before. This is the screen when no systems are currently selected:
image


Scenario: Check all new server certificates on the minions
When I check all certificates after renaming the server hostname

Scenario: Do some minimal smoke test on the renamed server
Given I am on the Systems overview page of this "sle_minion"
When I follow "Details" in the content area
Expand All @@ -40,3 +85,48 @@ Feature: Reconfigure the server's hostname
Scenario: Change hostname back and reboot server
When I change back the server's hostname
And I run spacewalk-hostname-rename command on the server

@proxy
Scenario: Copy the new server keys and configure the proxy
When I copy server's keys to the proxy
And I configure the proxy
Then I should see "proxy" via spacecmd
When I restart the "salt-minion" service on "proxy"
Then service "salt-minion" is active on "proxy"
When I restart the "salt-broker" service on "proxy"
Then service "salt-broker" is active on "proxy"

@proxy
Scenario: Apply high state on the proxy to populate new server CA
When I apply highstate on "proxy"

@sle_minion
Scenario: Apply high state on the SUSE Minion to populate new server CA
When I apply highstate on "sle_minion"

@ssh_minion
Scenario: Apply high state on the SUSE SSH Minion to populate new server CA
When I apply highstate on "ssh_minion"

@rhlike_minion
Scenario: Apply high state on the Red Hat-like Minion to populate new server CA
When I apply highstate on "rhlike_minion"

@deblike_minion
Scenario: Apply high state on the Debian-like Minion to populate new server CA
When I apply highstate on "deblike_minion"

@buildhost
Scenario: Apply high state on the build host to populate new server CA
When I apply highstate on "build_host"

@virthost_kvm
Scenario: Apply high state on the virthost to populate new server CA
When I apply highstate on "kvm_server"

@pxeboot_minion
Scenario: Apply high state on the PXE boot minion to populate new server CA
When I apply highstate on "pxeboot_minion"

Scenario: Check all new server certificates on the minions
When I check all certificates after renaming the server hostname
51 changes: 36 additions & 15 deletions testsuite/features/step_definitions/command_steps.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,16 +237,6 @@
raise 'Vendor change option not found in logs' unless return_code.zero?
end

When(/^I apply highstate on "([^"]*)"$/) do |host|
system_name = get_system_name(host)
if host.include? 'ssh_minion'
cmd = 'mgr-salt-ssh'
elsif host.include? 'minion' or host.include? 'build'
cmd = 'salt'
end
get_target('server').run_until_ok("#{cmd} #{system_name} state.highstate")
end

When(/^I wait until "([^"]*)" service is active on "([^"]*)"$/) do |service, host|
node = get_target(host)
cmd = "systemctl is-active #{service}"
Expand Down Expand Up @@ -1469,13 +1459,18 @@
When(/^I change the server's short hostname from hosts and hostname files$/) do
server_node = get_target('server')
old_hostname = server_node.hostname
new_hostname = old_hostname + '2'
new_hostname = old_hostname + '-renamed'
log "Old hostname: #{old_hostname} - New hostname: #{new_hostname}"
server_node.run("sed -i 's/#{old_hostname}/#{new_hostname}/g' /etc/hostname &&
hostname #{new_hostname} &&
echo '#{server_node.public_ip} #{server_node.full_hostname} #{old_hostname}' >> /etc/hosts &&
echo '#{server_node.public_ip} #{new_hostname}#{server_node.full_hostname.delete_prefix(server_node.hostname)} #{new_hostname}' >> /etc/hosts")
get_target('server', refresh: true) # This will refresh the attributes of this node
# This will refresh the attributes of this node
get_target('server', refresh: true)
hostname, _result = get_target('server').run('hostname')
hostname.strip!

raise "Wrong hostname after changing it. Is: #{hostname}, should be: #{new_hostname}" unless hostname == new_hostname
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe such basic checks would deserve a Then step of their own. It would make more explicit / readable that we do them.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we could do that. However, my main issue with this is, that our hostname rename and then changing the hostname back step, has a hardcoded new hostname variable in it. We do not define this new hostname in the step itself. Which is good, since across all our CIs and PR tests, the prefix of the hostname is different. Those are the reasons why I did not implement a new step of its own to check for the new hostname.


# Add the new hostname on controller's /etc/hosts to resolve in smoke tests
`echo '#{server_node.public_ip} #{new_hostname}#{server_node.full_hostname.delete_prefix(server_node.hostname)} #{new_hostname}' >> /etc/hosts`
Expand Down Expand Up @@ -1504,8 +1499,7 @@
end

# Update the server CA certificate since it changed, otherwise all API and browser uses will fail
update_ca('controller')
update_ca('proxy')
update_controller_ca

# Reset the API client to take the new CA into account
reset_api_client
Expand All @@ -1514,16 +1508,43 @@
raise 'Error in the output logs - see logs above' if out_spacewalk.include? 'No such file or directory'
end

When(/^I check all certificates after renaming the server hostname$/) do
# get server certificate serial to compare it with the other minions
command_server = "openssl x509 --noout --text -in /etc/pki/trust/anchors/LOCAL-RHN-ORG-TRUSTED-SSL-CERT | grep -A1 'Serial' | grep -v 'Serial'"
server_cert_serial, result_code = get_target('server').run(command_server)
server_cert_serial.strip!
log "Server certificate serial: #{server_cert_serial}"

raise 'Error getting server certificate serial!' unless result_code.zero?

command_minion = "openssl x509 --noout --text -in /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT | grep -A1 'Serial' | grep -v 'Serial'"
targets = %w[proxy sle_minion ssh_minion rhlike_minion deblike_minion build_host kvm_server]
targets.each do |target|
# get all defined minions from the environment variables and check their certificate serial
next unless ENV.key? ENV_VAR_BY_HOST[target]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can either do this or:
next if $node_by_host[target].nil?

Copy link
Member

@srbarrios srbarrios Oct 20, 2023 •
edited by nodeg
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But at a higher level, I think you can also do:

    $host_by_node.each do |node, _host|
      minion_cert_serial, result_code = node.run(command_minion)
      minion_cert_serial.strip!
      log "#{target} certificate serial: #{minion_cert_serial}"

      raise 'Error getting server certificate serial!' unless result_code.zero?
      raise "Error comparing #{target} certificate with server!" unless minion_cert_serial == server_cert_serial
    end

Bearing in mind that if you use this suggested loop, you will be iterating over all initialized twopences, so not only some particular targets.

Copy link
Member Author

@nodeg nodeg Oct 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. I will do some test run with this code.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This broke the code somehow.

  Scenario: Check all new server certificates on the minions         # features/secondary/srv_rename_hostname.feature:134
      This scenario ran at: 2023-10-20 16:53:27 +0200
    When I check all certificates after renaming the server hostname # features/step_definitions/command_steps.rb:1550
      Server certificate serial: 35:a1:54:c6:5e:c3:f5:f5:c6:d8:52:e6:f9:73:20:a2:d1:c5:75:83
      FAIL: openssl x509 --noout --text -in /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT | grep -A1 'Serial' | grep -v 'Serial' returned status code = 1.
      Output:
      Can't open /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT for reading, No such file or directory
      139797779679040:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT','r')
      139797779679040:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
      unable to load certificate
       (RuntimeError)
      ./features/support/lavanda.rb:177:in `run_local'
      ./features/support/lavanda.rb:155:in `run'
      ./features/step_definitions/command_steps.rb:1561:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/command_steps.rb:1560:in `each'
      ./features/step_definitions/command_steps.rb:1560:in `/^I check all certificates after renaming the server hostname$/'
      features/secondary/srv_rename_hostname.feature:135:in `I check all certificates after renaming the server hostname'
=> /var/log/rhn/rhn_web_ui.log

minion_cert_serial, result_code = get_target(target).run(command_minion)
minion_cert_serial.strip!
log "#{target} certificate serial: #{minion_cert_serial}"

raise 'Error getting server certificate serial!' unless result_code.zero?
raise "Error comparing #{target} certificate with server!" unless minion_cert_serial == server_cert_serial
end
end

When(/^I change back the server's hostname$/) do
server_node = get_target('server')
old_hostname = server_node.hostname
new_hostname = old_hostname.delete_suffix('2')
new_hostname = old_hostname.delete_suffix('-renamed')
log "Old hostname: #{old_hostname} - New hostname: #{new_hostname}"
server_node.run("sed -i 's/#{old_hostname}/#{new_hostname}/g' /etc/hostname &&
hostname #{new_hostname} &&
sed -i \'$d\' /etc/hosts &&
sed -i \'$d\' /etc/hosts")
get_target('server', refresh: true) # This will refresh the attributes of this node
hostname, _result = get_target('server').run('hostname')
hostname.strip!

raise "Wrong hostname after changing it. Is: #{hostname}, should be: #{new_hostname}" unless hostname == new_hostname

# Cleanup the temporary entry in /etc/hosts on the controller
`sed -i \'$d\' /etc/hosts`
Expand Down
11 changes: 11 additions & 0 deletions testsuite/features/step_definitions/salt_steps.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,3 +578,14 @@ def pillar_get(key, minion)
cleanup = %(salt #{system_name} state.apply util.mgr_switch_to_venv_minion pillar='{"mgr_purge_non_venv_salt_files": True, "mgr_purge_non_venv_salt": True}')
get_target('server').run(cleanup, check_errors: true, verbose: true)
end

When(/^I apply highstate on "([^"]*)"$/) do |host|
system_name = get_system_name(host)
if host.include? 'ssh_minion'
cmd = 'mgr-salt-ssh'
elsif host.include? 'minion' or host.include? 'build' or host.include? 'proxy'
cmd = 'salt'
end
log "#{cmd} #{system_name} state.highstate"
get_target('server').run_until_ok("#{cmd} #{system_name} state.highstate")
end
19 changes: 5 additions & 14 deletions testsuite/features/support/commonlib.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,23 +418,14 @@ def file_inject(node, local_file, remote_file)
end

# This function updates the server certificate on the controller node
def update_ca(node)
def update_controller_ca
server_ip = get_target('server').public_ip
server_name = get_target('server').full_hostname

case node
when 'proxy'
command = "wget http://#{server_ip}/pub/RHN-ORG-TRUSTED-SSL-CERT -O /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT; " \
'update-ca-certificates;'
get_target('proxy').run('rm /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT', verbose: true)
get_target('proxy').run(command, verbose: true)
else
# controller
puts `rm /etc/pki/trust/anchors/*;
wget http://#{server_ip}/pub/RHN-ORG-TRUSTED-SSL-CERT -O /etc/pki/trust/anchors/#{server_name}.cert &&
update-ca-certificates &&
certutil -d sql:/root/.pki/nssdb -A -t TC -n "susemanager" -i /etc/pki/trust/anchors/#{server_name}.cert`
end
puts `rm /etc/pki/trust/anchors/*;
wget http://#{server_ip}/pub/RHN-ORG-TRUSTED-SSL-CERT -O /etc/pki/trust/anchors/#{server_name}.cert &&
update-ca-certificates &&
certutil -d sql:/root/.pki/nssdb -A -t TC -n "susemanager" -i /etc/pki/trust/anchors/#{server_name}.cert`
end

# This functions checks if the channel has been synced
Expand Down
Loading