Foreman Maintain

# cat inventory
[foreman]
<hostname>

# ansible foreman -i inventory -m command -a "./foreman_maintain/bin/foreman-maintain upgrade run --target-version 6.2.z -y
" 
qe-sat62-rhel7.satqe.lab.eng.rdu2.redhat.com | FAILED | rc=1 >>
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer:                                                         [OK]
--------------------------------------------------------------------------------
Procedures::Packages::Install:
                                                                      [FAIL]
Failed executing yum -y install hdparm fio, exit status 256
--------------------------------------------------------------------------------
Scenario [preparation steps required to run the next scenarios] failed.

The following steps ended up in failing state:

  [packages-install]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="packages-install" 

Running Checks before upgrading to Satellite 6.2.z
================================================================================
Check for verifying syntax for ISP DHCP configurations:               [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using hammer ping:             [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:
================================================================================
 Package         Arch        Version              Repository               Size
================================================================================
Installing:
 fio             x86_64      3.1-2.el7            rhel-7-server-rpms      451 k
 hdparm          x86_64      9.43-5.el7           rhel-7-server-rpms       83 k
Installing for dependencies:
 libibverbs      x86_64      15-6.el7             rhel-7-server-rpms      224 k
 libpmem         x86_64      1.3-3.el7            rhel-7-server-rpms       36 k
 libpmemblk      x86_64      1.3-3.el7            rhel-7-server-rpms       54 k
 librados2       x86_64      1:0.94.5-2.el7       rhel-7-server-rpms      1.7 M
 librbd1         x86_64      1:0.94.5-2.el7       rhel-7-server-rpms      1.8 M
 librdmacm       x86_64      15-6.el7             rhel-7-server-rpms       61 k
 rdma-core       x86_64      15-6.el7             rhel-7-server-rpms       48 k

Transaction Summary
================================================================================
Install  2 Packages (+7 Dependent packages)

Total download size: 4.4 M
Installed size: 13 M
                                [OK]
--------------------------------------------------------------------------------
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for old tasks in paused/stopped state:                          [OK]
--------------------------------------------------------------------------------
Check for pending tasks which are safe to delete:                     [OK]
--------------------------------------------------------------------------------
Check for tasks in planning state:                                    [OK]
--------------------------------------------------------------------------------
Check for recommended disk speed of pulp, mongodb, pgsql dir.:
\ [Speed check In-Progress] device: /dev/mapper/vg_dhcp201-lv_root
Installed:
  fio.x86_64 0:3.1-2.el7               hdparm.x86_64 0:9.43-5.el7

Dependency Installed:
  libibverbs.x86_64 0:15-6.el7          libpmem.x86_64 0:1.3-3.el7
  libpmemblk.x86_64 0:1.3-3.el7         librados2.x86_64 1:0.94.5-2.el7
  librbd1.x86_64 1:0.94.5-2.el7         librdmacm.x86_64 0:15-6.el7
  rdma-core.x86_64 0:15-6.el7

- Finished

Disk speed : 134 MB/sec                                               [OK]
--------------------------------------------------------------------------------
Validate availability of repositories:
| Validating availability of repositories for 6.2                     [OK]
--------------------------------------------------------------------------------

The pre-upgrade checks indicate that the system is ready for upgrade.
It's recommended to perform a backup at this stage.
Running Procedures before migrating to Satellite 6.2.z
================================================================================
Turn on maintenance mode:                                             [OK]
--------------------------------------------------------------------------------
disable active sync plans:
\ Total 0 sync plans are now disabled.                                [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):

postgresql, mongod, squid, qpidd, qdrouterd, tomcat, pulp_streamer, smart_proxy_dynflow_core, pulp_resource_manager, pulp_celerybeat, pulp_workers,
 foreman-proxy, httpd, foreman-tasks

| Services are handled by katello-service in Satellite versions 6.2 and earlier. Redirecting to:
katello-service stop

/ Redirecting to /bin/systemctl stop foreman-tasks.service

Redirecting to /bin/systemctl stop httpd.service

Redirecting to /bin/systemctl stop pulp_workers.service

Redirecting to /bin/systemctl stop foreman-proxy.service

Redirecting to /bin/systemctl stop pulp_streamer.service

Redirecting to /bin/systemctl stop pulp_resource_manager.service

Redirecting to /bin/systemctl stop pulp_celerybeat.service

Redirecting to /bin/systemctl stop smart_proxy_dynflow_core.service

Redirecting to /bin/systemctl stop tomcat.service

Redirecting to /bin/systemctl stop squid.service

Redirecting to /bin/systemctl stop qdrouterd.service

Redirecting to /bin/systemctl stop qpidd.service

Redirecting to /bin/systemctl stop postgresql.service

Redirecting to /bin/systemctl stop mongod.service

Success!                                                              [OK]
--------------------------------------------------------------------------------

Running Migration scripts to Satellite 6.2.z
================================================================================
Setup repositories:
- Configuring repositories for 6.2                                    [OK]
--------------------------------------------------------------------------------
Update package(s) :
                                                                      [FAIL]
Failed executing yum -y clean all, exit status 256
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.2.z] failed.

The following steps ended up in failing state:

  [packages-update]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="packages-update"non-zero return code

and foreman-maintain.log shows

D, [2018-04-16 05:25:54-0400 #12546] DEBUG -- : Running command yum -y install hdparm fio with stdin nil
D, [2018-04-16 05:25:54-0400 #12546] DEBUG -- : output of the command:

E, [2018-04-16 05:25:54-0400 #12546] ERROR -- : Failed executing yum -y install hdparm fio, exit status 256 (ForemanMaintain::Error::ExecutionError
)
/root/foreman_maintain/lib/foreman_maintain/utils/command_runner.rb:50:in `execution_error'
/root/foreman_maintain/lib/foreman_maintain/concerns/system_helpers.rb:66:in `execute!'
/root/foreman_maintain/lib/foreman_maintain/concerns/system_helpers.rb:113:in `packages_action'
/root/foreman_maintain/definitions/procedures/packages/install.rb:9:in `run'
/root/foreman_maintain/lib/foreman_maintain/executable.rb:120:in `__run__'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:79:in `block (2 levels) in run'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:103:in `capture_errors'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:78:in `block in run'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:97:in `with_metadata_calculation'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:77:in `run'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:109:in `run_step'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:98:in `run_steps'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:91:in `execute_scenario_steps'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:45:in `run_scenario'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:88:in `block in execute_scenario_steps'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:88:in `each'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:88:in `execute_scenario_steps'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:45:in `run_scenario'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:138:in `block in run_phase'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:183:in `with_non_empty_scenario'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:134:in `run_phase'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:82:in `block in run'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:77:in `each'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:77:in `run'
/root/foreman_maintain/lib/foreman_maintain/cli/upgrade_command.rb:80:in `execute'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/root/foreman_maintain/lib/foreman_maintain/cli.rb:23:in `run'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
./foreman_maintain/bin/foreman-maintain:12:in `<main>'

D, [2018-04-16 05:27:04-0400 #12546] DEBUG -- : Running command yum -y clean all with stdin nil
D, [2018-04-16 05:27:04-0400 #12546] DEBUG -- : output of the command:

E, [2018-04-16 05:27:04-0400 #12546] ERROR -- : Failed executing yum -y clean all, exit status 256 (ForemanMaintain::Error::ExecutionError)
/root/foreman_maintain/lib/foreman_maintain/utils/command_runner.rb:50:in `execution_error'
/root/foreman_maintain/lib/foreman_maintain/concerns/system_helpers.rb:66:in `execute!'
/root/foreman_maintain/lib/foreman_maintain/concerns/system_helpers.rb:121:in `clean_all_packages'
/root/foreman_maintain/definitions/procedures/packages/update.rb:10:in `run'
/root/foreman_maintain/lib/foreman_maintain/executable.rb:120:in `__run__'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:79:in `block (2 levels) in run'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:103:in `capture_errors'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:78:in `block in run'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:97:in `with_metadata_calculation'
/root/foreman_maintain/lib/foreman_maintain/runner/execution.rb:77:in `run'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:109:in `run_step'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:98:in `run_steps'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:91:in `execute_scenario_steps'
/root/foreman_maintain/lib/foreman_maintain/runner.rb:45:in `run_scenario'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:138:in `block in run_phase'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:183:in `with_non_empty_scenario'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:134:in `run_phase'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:82:in `block in run'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:77:in `each'
/root/foreman_maintain/lib/foreman_maintain/upgrade_runner.rb:77:in `run'
/root/foreman_maintain/lib/foreman_maintain/cli/upgrade_command.rb:80:in `execute'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/root/foreman_maintain/lib/foreman_maintain/cli.rb:23:in `run'
/usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
./foreman_maintain/bin/foreman-maintain:12:in `<main>'
I, [2018-04-16 05:27:04-0400 #12546]  INFO -- : foreman-maintain command finished with

Additional info : Running foreman-maintain upgrade works fine when directly run on target system.