Restoring customerDB failed at step: Reset katello index

[sghai]

TASK [satellite-clone : Reset katello index - Note that this might take hours] *
skipping: [SERVER]

TASK [satellite-clone : Reset katello index - Note that this might take hours] *
fatal: [SERVER]: FAILED! => {"changed": true, "cmd": ["foreman-rake", "katello:reimport", "--trace"], "delta": "0:02:34.273112", "end": "2017-04-20 05:04:39.850815", "failed": true, "rc": 1, "start": "2017-04-20 05:02:05.577703", "stderr": "** Invoke katello:reimport (first_time)\n** Invoke environment (first_time)\n** Execute environment\n/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES\n/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here\n** Invoke katello:check_ping (first_time)\n** Execute katello:check_ping\n** Execute katello:reimport\nrake aborted!\nNoMethodError: undefined method `each' for nil:NilClass\n/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/activation_key.rb:32:in `get_key_pools'\n/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/activation_key.rb:40:in `import_pools'\n/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/reimport.rake:37:in `block (3 levels) in <top (required)>'\n/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'\n/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'\n/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/reimport.rake:36:in `block (2 levels) in <top (required)>'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'\n/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'\n/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'\n/opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'\nTasks: TOP => katello:reimport", "stdout": "Importing Katello::Erratum\nImporting Katello::PackageGroup\nImporting Katello::PuppetModule\nImporting Katello::Rpm\nImporting Katello::Subscription\nImporting Katello::Pool\nImporting Katello::DockerManifest\nImporting Katello::DockerTag\nImporting Katello::ContentViewPuppetEnvironment\nImporting Activation Key Subscriptions", "stdout_lines": ["Importing Katello::Erratum", "Importing Katello::PackageGroup", "Importing Katello::PuppetModule", "Importing Katello::Rpm", "Importing Katello::Subscription", "Importing Katello::Pool", "Importing Katello::DockerManifest", "Importing Katello::DockerTag", "Importing Katello::ContentViewPuppetEnvironment", "Importing Activation Key Subscriptions"], "warnings": []}
	to retry, use: --limit @/home/jenkins/workspace/satellite6_db_upgrade_migrate/satellite-clone/satellite-clone-playbook.retry

[sghai]

Its not consistently appearing but 2 times out of 5. Earlier I reported this on email and there we ran # foreman-rake katello:reimport --trace and this time again I re-executed # foreman-rake katello:reimport --trace after failure and it was completed successfully. Not sure if its just a timing issue ?

[sghai]

Please note that it is appearing w/ 1.0.1 release

[johnpmitsch]

Note to John: check if this is still an issue